Method for reasoning text content through generative artificial intelligence video based on time axis

Abstract

The invention discloses a generation type artificial intelligence video text content reasoning method based on a time axis, and relates to the technical field of artificial intelligence. A session is established between a user side and an algorithm side service, and the user side pushes a video and a problem to the algorithm side service; obtaining streaming video data, decoding the streaming video data in real time, and slicing the streaming video data according to a time axis; submitting the picture content to an LLM reasoning service, obtaining text data after picture reasoning and a generated timestamp, and storing the text data and the generated timestamp into a data mirror image database; the user side conversations convert voice into text content, and tuple data is formed by combining the triggering time and the current video picture frame; a complete frame sequence data set is obtained from the tuple and the video slice cache set according to the sequence of timestamps, and the set retains contexts required by video dialogues; and the LLM reasoning service feeds back response data information according to the context. According to the method, the video stream is decoded in real time and sliced according to the time axis, and the LLM reasoning service is combined, so that instant analysis and textualized presentation of the video content are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a method for timeline-based generative artificial intelligence video inference text content. Background Art

[0002] In LLM, it is difficult to infer text content from videos. On the one hand, the amount of data generated by the video is large, and on the other hand, it takes a lot of time to decompress the video. It is difficult to generate text content from the video in real time, and the user experience will be very poor. In addition, it is difficult to extract features from the video. Video conversations require context, and the traditional video frame method cannot solve the context problem. The LLM inference model cannot correctly perceive the user's specific real-time behavior.

[0003] Therefore, a method of generative artificial intelligence video inference text content based on timeline has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide a method for timeline-based generative artificial intelligence video reasoning text content, which solves the problems of response, correctness, context, etc. of streaming video from the perspective of video slice cache and data tuple sequence.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: a method for generating artificial intelligence video inference text content based on timeline, comprising the following steps:

[0006] S1. The user terminal establishes a session N0 with the algorithm service. The user terminal pushes the video and questions to the algorithm service. The session N0 is maintained until the user exits.

[0007] S2, obtain streaming video data for real-time decoding, slice it according to the time axis, and obtain a set of discontinuous and continuous pictures in fixed time units;

[0008] S3, submit the image content to the LLM reasoning service, and obtain the text data S0 and the generated timestamp T0 after the reasoning image;

[0009] S4, storing the text data S0 and timestamp T0 into the data mirror database, the retrieval ID of the storage method is N0 of this session, and subsequently obtaining all the slice data sets in this session through N0;

[0010] S5. Obtain a data set D0 in session N0 through steps S2-S4, where the data set D0 is a video slice cache set for subsequent video and text processing;

[0011] S6, the user end dialogue converts the voice into text content, combines the trigger time and the current video frame, and forms a tuple data of M0{text, time, picture};

[0012] S7, combine the M0 tuple with the D0 video slice cache set in the order of timestamps to obtain a complete frame sequence data set, which retains the context V0 required for the video conversation;

[0013] S8. Submit the context V0 obtained in S7 to the LLM reasoning service, and the LLM reasoning service feeds back response data information according to the context.

[0014] The advantages of the present invention compared with the prior art are:

[0015] 1. Real-time and high efficiency: The present invention realizes instant analysis and textual presentation of video content by real-time decoding of video streams and slicing them according to the time axis, combined with LLM reasoning services.

[0016] 2. Intelligence and accuracy: The present invention utilizes the powerful reasoning ability of LLM to deeply understand and describe the video content, thereby improving the accuracy and intelligence of text conversion.

[0017] 3. Flexibility and scalability: The present invention supports flexible establishment and termination of user sessions, as well as construction and updating of video slice cache sets, facilitating subsequent processing and expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for inferring text content from a timeline-based generative artificial intelligence video of the present invention. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0022] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0023] Combined with Figure 1 The specific implementation process of the method for generating artificial intelligence video inference text content based on timeline of the present invention is as follows:

[0024] A method for generating artificial intelligence video inference text content based on a timeline, comprising the following steps:

[0025] S1. The user terminal establishes a session N0 with the algorithm service. The user terminal pushes the video and questions to the algorithm service. The session N0 is maintained until the user exits.

[0026] The user side and the algorithm side service communicate via HTTP. Each user will establish an independent session and use a unique ID as an identifier.

[0027] The client will interact with the transmission of videos and questions through the session, and the remote service will receive the data request from the client.

[0028] S2, obtain streaming video data for real-time decoding, slice it according to the time axis, and obtain a set of discontinuous and continuous pictures in fixed time units;

[0029] The client will decode the streaming video in real time locally and convert it into frame-by-frame images;

[0030] The video frames are sliced ​​at fixed time intervals of 1 second, and each slice contains several frames of images. This process generates an intermittent but continuous sequence of images.

[0031] S3, submit the image content to the LLM reasoning service, and obtain the text data S0 and the generated timestamp T0 after the reasoning image;

[0032] Submit the image content to the LLM inference service that supports image-to-text conversion through HTTP request.

[0033] S4, storing the text data S0 and timestamp T0 into the data mirror database, the retrieval ID of the storage method is N0 of this session, and subsequently obtaining all the slice data sets in this session through N0;

[0034] The table structure of the database includes session ID, slice timestamp, and inference text.

[0035] S5. Obtain a data set D0 in session N0 through steps S2-S4, where the data set D0 is a video slice cache set for subsequent video and text processing;

[0036] S6, the user end dialogue converts the voice into text content, combines the trigger time and the current video frame, and forms a tuple data of M0{text, time, picture};

[0037] The user's conversation speech is synthesized into an audio file, and the audio file is pushed to an online audio-to-text service via Http, thereby obtaining the text information after the audio is converted.

[0038] S7, combine the M0 tuple with the D0 video slice cache set in the order of timestamps to obtain a complete frame sequence data set, which retains the context V0 required for the video conversation;

[0039] S8. Submit the context V0 obtained in S7 to the LLM reasoning service, and the LLM reasoning service feeds back response data information according to the context.

[0040] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A method for timeline-based generative artificial intelligence video reasoning text content, characterized by: The following steps are included: S1. The user terminal establishes a session N0 with the algorithm service. The user terminal pushes the video and questions to the algorithm service. The session N0 is maintained until the user exits. S2, obtain streaming video data for real-time decoding, slice it according to the time axis, and obtain a set of discontinuous and continuous pictures in fixed time units; S3, submit the image content to the LLM reasoning service, and obtain the text data S0 and the generated timestamp T0 after the reasoning image; S4, storing the text data S0 and timestamp T0 into the data mirror database, the retrieval ID of the storage method is N0 of this session, and subsequently obtaining all the slice data sets in this session through N0; S5. Obtain a data set D0 in session N0 through steps S2-S4, where the data set D0 is a video slice cache set for subsequent video and text processing; S6, the user end dialogue converts the voice into text content, combines the trigger time and the current video frame, and forms a tuple data of M0{text, time, picture}; S7, combine the M0 tuple with the D0 video slice cache set in the order of timestamps to obtain a complete frame sequence data set, which retains the context V0 required for the video conversation; S8. Submit the context V0 obtained in S7 to the LLM reasoning service, and the LLM reasoning service feeds back response data information according to the context.

2. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 1, characterized in that: In S1, the user side and the algorithm side service communicate via HTTP. Each user will establish an independent session and use a unique ID as an identifier.

3. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 2, characterized in that: In S1, the user will interact with the transmission of videos and questions through the session, and the remote service will receive the data request from the user.

4. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 3, characterized in that: In S2, the client will decode the streaming video in real time locally and convert it into frame-by-frame images.

5. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 4, characterized in that: In S2, the video frames are sliced ​​at fixed time intervals of 1 second, and each slice contains several frames of images. This process generates an intermittent but continuous sequence of pictures.

6. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 5, characterized in that: Submit the image content to the LLM inference service that supports image-to-text conversion through HTTP request in S3.

7. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 6, characterized in that: The table structure of the database in S4 includes session ID, slice timestamp, and inference text.

8. The method of timeline-based generative artificial intelligence video reasoning text content according to claim 7, characterized in that: In S6, the user's conversation speech is synthesized into an audio file, and the audio file is pushed to the online audio-to-text service via Http. Thereby obtaining the text information after the audio is converted.

Images