A method for generating information based on multimodal model and related equipment

Abstract

The present application provides an information generation method based on a multimodal model and related equipment, the method comprising: extracting features from acquired video data to obtain video features, the video features including static features and dynamic features; determining a target prompt in a preset database according to the acquired text data, the preset database storing multiple prompts, the target prompt being a prompt matching the text data in the preset database; mapping the multimodal data to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, the multimodal data including video features and target prompts, the video content information being information describing the video data in the form of dialogue text. The deep fusion between different modalities at the semantic level is achieved, so that the video features and the target prompts in text form can be directly semantically associated and interacted, thereby improving the accuracy of the generated information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information generation method based on a multimodal model and related equipment. Background Art

[0002] With the explosive growth of video data, it is increasingly important to effectively understand video content. However, when dealing with multimodal video content understanding, there is currently a problem of inconsistent visual feature representation, resulting in poor accuracy of the generated information.

[0003] In today's digital age, video data is growing explosively, playing an increasingly critical role in many fields such as entertainment, education, security, and healthcare. This makes effective understanding of video content a challenging but crucial task. Despite numerous studies and practices, it still faces many difficulties. Among them, the problem of inconsistent visual feature representation is particularly prominent. Different video content may contain a variety of visual elements. Traditional methods often lack unified standards and effective integration methods when processing these visual elements, resulting in poor accuracy in generating information related to video content. Summary of the invention

[0004] The main purpose of this application is to provide an information generation method and related equipment based on a multimodal model, aiming to solve the problem of poor accuracy of information generated by existing information generation methods.

[0005] To achieve the above objectives, the present application provides a method for generating information based on a multimodal model, the method comprising:

[0006] Extracting features from the acquired video data to obtain video features, wherein the video features include static features and dynamic features, wherein the static features include color features, texture features, and shape features, and the dynamic features include camera parameter features, as well as velocity vector features and position vector features of a target object in the video data;

[0007] According to the acquired text data, a target prompt word may be determined in a preset database, wherein a plurality of prompt words are stored in the preset database, and the target prompt word is a prompt word in the preset database that matches the text data;

[0008] Based on a pre-trained multimodal model, multimodal data are mapped to a shared semantic embedding space to generate video content information, wherein the multimodal data includes the video features and the target prompts, and the video content information is information that describes the video data in the form of dialogue text.

[0009] Optionally, the method further comprises:

[0010] Extracting features from at least a portion of the image frames in the acquired video data to obtain image features;

[0011] In the case where the video data includes audio data, extracting features from the audio data to obtain audio features;

[0012] The multimodal data also includes the image features and the audio features.

[0013] Optionally, the extracting features from the acquired video data to obtain video features includes:

[0014] Performing frame rate conversion processing and size conversion processing on the video data to obtain a plurality of image frames, wherein intervals between adjacent frames in the plurality of image frames are the same, and the intervals match a frame rate corresponding to a first preset video type, and resolutions of the plurality of image frames are the same, and the resolutions match a resolution corresponding to the first preset video type, and the first preset video type is a video type determined according to key frames in the video data;

[0015] The multiple image frames are input into a video encoder to obtain the video features.

[0016] Optionally, determining a target prompt in a preset database according to the acquired text data may include:

[0017] Vectorizing the text data to obtain text vector information;

[0018] The target prompt is determined in the preset database according to the text vector information.

[0019] Optionally, before mapping the multimodal data to a shared semantic embedding space based on the pre-trained multimodal model to generate video content information, the method further includes:

[0020] Acquire training data, wherein the training data includes a sample video set, a sample text set, a sample audio set, and corresponding information annotations;

[0021] Obtaining first model parameters corresponding to the first model after training based on the training data, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, the preset initial model includes a plurality of neural network layers, and the preset initial model is used to learn and process the fused multimodal features, and i is a positive integer;

[0022] The model parameters of the first model are updated based on the first model parameters to obtain the multimodal model, wherein the multimodal model is a model obtained by updating the preset initial model at the i+1th iteration.

[0023] Optionally, the multimodal data is mapped to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, including:

[0024] Calculating the correlation weights between each modal feature and other modal features in the multimodal data to dynamically allocate attention resources to the multimodal data;

[0025] A sum operation is performed on the multimodal data according to the attention resource to obtain video content information, wherein the video content information includes a fused multimodal feature vector.

[0026] In order to solve the above technical problems, the embodiment of the present application further provides an information generation device based on a multimodal model, the device comprising:

[0027] A first feature extraction module is used to extract features from the acquired video data to obtain video features, wherein the video features include static features and dynamic features, wherein the static features include color features, texture features, and shape features, and the dynamic features include camera parameter features, and velocity vector features and position vector features of a target object in the video data;

[0028] A determination module, configured to determine a target prompt in a preset database according to the acquired text data, wherein the preset database stores a plurality of prompts, and the target prompt is a prompt in the preset database that matches the text data;

[0029] A generation module is used to map multimodal data to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, wherein the multimodal data includes the video features and the target prompts, and the video content information is information that describes the video data in the form of dialogue text.

[0030] In order to solve the above technical problems, the embodiment of the present application further provides an electronic device, which adopts the following technical solution:

[0031] The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the information generation methods based on a multimodal model proposed in the embodiments of the present application are implemented.

[0032] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0033] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of any one of the information generation methods based on a multimodal model proposed in the embodiments of the present application are implemented.

[0034] In order to solve the above technical problems, the embodiment of the present application also provides a computer program product, which adopts the following technical solution:

[0035] The computer program product includes computer instructions, which, when executed by a processor, implement the steps of any one of the information generation methods based on a multimodal model proposed in the embodiments of the present application.

[0036] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0037] The present application provides an information generation method based on a multimodal model, which extracts features from video data, covers static and dynamic features, and provides an accurate dynamic information basis for subsequent information generation. Compared with the feature extraction method that lacks a unified standard in traditional methods, it greatly improves the depth and accuracy of understanding the video content. According to the text data, the target prompt is determined in the preset database, which can provide effective guidance information for the multimodal model. The text data may include the user's questions, concerns or specific requirements for the video content, and the target prompt is to select the content that best matches the text data from the preset rich prompt library, which can help the model understand the user's intention to enter the text more accurately. Different text data are matched to different target prompts, and the strategy for understanding the video content can be flexibly adjusted according to the specific text input. Compared with the traditional method, the model is no longer limited to a single, fixed video content understanding mode, but can understand the video content from multiple dimensions with the guidance of the target prompt according to the different needs of the user and the diversity of the input text, and then generate video content information that is more in line with the user's expectations, thereby improving the accuracy and practicality of information generation. The pre-trained multimodal model maps video features and target prompts to a shared semantic embedding space, successfully achieving deep fusion between different modalities at the semantic level, allowing video features and target prompts in text form to directly associate and interact with each other, overcoming the problem of difficult effective integration of different modal information in traditional methods. It can also make full use of various information in video features and the guidance and contextual information provided by target prompts to generate detailed, accurate and logical descriptions in the form of dialogue text. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0040] Figure 2 It is one of the flow charts of the information generation method based on the multimodal model provided in the embodiment of the present application;

[0041] Figure 3 This is the second flowchart of the information generation method based on the multimodal model provided in the embodiment of the present application;

[0042] Figure 4 This is the third flowchart of the information generation method based on the multimodal model provided in the embodiment of the present application;

[0043] Figure 5 This is the fourth flowchart of the information generation method based on the multimodal model provided in the embodiment of the present application;

[0044] Figure 6 This is the fifth flowchart of the information generation method based on the multimodal model provided in the embodiment of the present application;

[0045] Figure 7 It is a structural schematic diagram of an information generation device based on a multimodal model provided in an embodiment of the present application;

[0046] Figure 8 This is a basic structural block diagram of the electronic device of this embodiment. DETAILED DESCRIPTION

[0047] The information generation method based on the multimodal model provided in the embodiment of the present application is applied to the information generation device based on the multimodal model. Unless otherwise defined, all technical and scientific terms used in this article have the same meaning as those generally understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned figures are used to distinguish different objects, not to describe a specific order.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0050] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0051] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. In some embodiments, various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social online platform software, etc. The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) players, MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop portable computers and desktop computers, etc.

[0052] It should be noted that the information generation method based on the multimodal model provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the information generation device based on the multimodal model is generally arranged in the server / terminal device.

[0053] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0054] Continue to refer Figure 2, shows one of the flow charts of a method for generating information based on a multimodal model proposed in the present application. The method for generating information based on a multimodal model provided in the embodiment of the present application can be applied to a server, such as Figure 2 As shown, a method for generating information based on a multimodal model provided in an embodiment of the present application includes the following steps:

[0055] S210, extracting features from the acquired video data to obtain video features, where the video features include static features and dynamic features;

[0056] In this step, feature extraction is performed on the acquired video data, and static features and dynamic features can be extracted for each image frame in the video data. The static features include color features, texture features and shape features, and the dynamic features include camera parameter features, as well as speed vector features and position vector features of the target object in the video data. In terms of static feature extraction, advanced image analysis algorithms, such as convolutional neural network models based on deep learning, are used to accurately extract the color features of key frames, and quantify the color information into specific color space vectors for subsequent comparison and calculation; for texture features, algorithms such as grayscale co-occurrence matrix or local binary pattern are used to describe the texture information of the image and obtain texture feature vectors; shape features are extracted through edge detection algorithms and shape descriptors to obtain feature vectors that can characterize the shape of the object. In the case of dynamic feature extraction, the optical flow method is used to analyze the pixel motion between adjacent frames, so as to obtain information such as the motion trajectory, speed and direction of the object in the lens, and at the same time, the motion parameters of the camera are recorded, such as the angle of the panning lens, the distance and speed of the push-pull, and the target object being tracked, and these dynamic features are quantized and encoded to form a dynamic feature vector in a unified format.

[0057] Among them, the dynamic feature vector can accurately describe the camera parameter features in the video data and the velocity vector features and position vector features of the target object. With this information, the model can deeply understand the dynamic changes in the video, such as the camera movement mode (panning, pushing, pulling, tracking, etc.) and the movement trajectory, speed change and position relationship of the target object in the video. This is crucial for identifying event types and scene transitions in the video. For example, in surveillance videos, by analyzing the dynamic feature vector, it is possible to determine whether an object has entered the surveillance area, whether the object's movement direction and speed are abnormal, thereby realizing the detection and early warning of abnormal behavior. In the subsequent multimodal information fusion process, the dynamic feature vector is interactively fused with the language mode represented by the text data (such as the target prompt). By establishing an association with the text features in the shared semantic embedding space, the dynamic information in the video is passed to the language mode, so that the text can better understand and explain the dynamic events in the video. For example, when the text prompt asks about the movement status of an object in the video, the dynamic feature vector can provide accurate motion information, which is combined with the text features to generate a more detailed and accurate answer.

[0058] In addition, if Figure 3 As shown, before performing video processing on the acquired video data, it is determined whether the acquired video data is in a legal format, so as to determine whether a dynamic feature vector of a unified format can be formed in the subsequent processing process; before performing text processing on the acquired text data, the user's input, that is, the acquired text data, can be vectorized, the purpose of which is to convert the text data into a vector representation so that the computer can process it, and find a suitable prompt from the prompt template library (that is, the preset database) through the retrieval module as the input of the multimodal model. Then, based on the pre-trained multimodal model, the video features and text features are fused, so that the multimodal information can be directly understood and aligned at the semantic level, thereby better fusing the video features and text features. Finally, after aligning the multimodal features, the powerful content understanding ability of the multimodal model can be used to generate text output. At the same time, the content can also form contextual content with the question and continue to be input into the model, so as to output text with the content that can understand multiple rounds of dialogue.

[0059] S220, determining a target prompt in a preset database according to the acquired text data, wherein the preset database stores a plurality of prompts, and the target prompt is a prompt in the preset database that matches the text data;

[0060] In this step, a preset database is pre-built and multiple prompts are stored in the preset database, thereby realizing an efficient index structure to accelerate the search process. Specifically, an inverted index technology can be used, that is, each word is associated with a prompt document containing the word, and each prompt can be regarded as a document to establish a mapping relationship. In this way, when searching for prompts containing specific words, the relevant prompt documents can be quickly located.

[0061] Among them, the prompt can be a prompt, which can clearly specify the format of the model output information to meet the needs of specific application scenarios. And when the user asks about the behavior of a certain character in the video, the prompt may contain content such as "analyze the action details, behavior purpose and interaction with the surrounding environment of the character in the video", guiding the model to focus on the visual information and text description related to the character's behavior, avoiding deviations or misunderstandings when the model understands the problem. In addition, the questions entered by the user may not contain all the necessary information, and the prompt can supplement this missing key information. For example, for a question involving the judgment of the attributes of an object in a specific scene, the prompt can provide background knowledge or relevant contextual information of the scene, so that the model can understand the problem more comprehensively and generate answers more accurately.

[0062] Then, the best matching prompt template (i.e., target prompt) can be determined in the preset database by using the retrieval enhancement method to obtain and input the user (i.e., text data) to enhance the output effect of the model, and finally complete a multimodal model application of video content understanding based on the form of dialogue. Exemplarily, the acquired text data can be preprocessed first to remove noise information in the text, such as extra spaces, wrong punctuation marks, special characters, etc., to ensure that the text format is consistent. Then the preprocessed text data is segmented using a natural language processing tool to split the text into individual words or vocabulary units, and stem extraction or word form restoration is performed to obtain the basic form of the word. Then, based on the keywords, a search is performed in the inverted index to obtain a candidate set of prompts containing these keywords. Among them, the candidate prompts in the candidate set of prompts can be sorted according to the calculated similarity score, and the prompts with high similarity scores are placed in front. In this way, the candidate prompts with the highest scores can be determined as target prompts.

[0063] S230, mapping the multimodal data to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, wherein the multimodal data includes the video features and the target prompts, and the video content information is information that describes the video data in the form of dialogue text.

[0064] In this step, the multimodal data types include video features (static and dynamic features) and target prompts (text data), which describe the video content from different perspectives. Through the processing of the multimodal model, a more comprehensive and in-depth understanding of the video and accurate information generation are achieved. Based on the pre-trained multimodal model, the multimodal data is mapped to a shared semantic embedding space, that is, the multimodal data is mapped to the shared semantic embedding space through a contrastive learning mechanism, so as to better integrate video features and text features, and improve the performance of the multimodal model in tasks such as video content understanding and information generation. In the shared semantic embedding space, the language modality is directly used as a link to avoid the semantic information attenuation caused by the two-step conversion process in the traditional image mediation method. Because the language modality itself has rich semantic expression capabilities and can convey the meaning of information more directly, the semantic information of the original modality can be better preserved in the process of mapping to the shared semantic embedding space.

[0065] In this embodiment, feature extraction is performed on video data, covering static and dynamic features, providing an accurate dynamic information basis for subsequent information generation, and compared with the feature extraction method lacking a unified standard in the traditional method, the depth and accuracy of understanding the video content are greatly improved. According to the text data, the target prompt is determined in the preset database, which can provide effective guidance information for the multimodal model. The text data can include the user's questions, concerns or specific requirements for the video content, and the target prompt is to screen out the content that best matches the text data from the preset rich prompt library, which can help the model to more accurately understand the intention of the user input text. Different text data are matched to different target prompts, and the understanding strategy of the video content can be flexibly adjusted according to the specific text input. Compared with the traditional method, the model is no longer limited to a single, fixed video content understanding mode, but can understand the video content from multiple dimensions according to the different needs of the user and the diversity of the input text, with the guidance of the target prompt, and then generate video content information that is more in line with the user's expectations, improving the accuracy and practicality of information generation. The pre-trained multimodal model maps video features and target prompts to a shared semantic embedding space, successfully achieving deep fusion between different modalities at the semantic level, allowing video features and target prompts in text form to directly associate and interact with each other, overcoming the problem of difficult effective integration of different modal information in traditional methods. It can also make full use of various information in video features and the guidance and contextual information provided by target prompts to generate detailed, accurate and logical descriptions in the form of dialogue text.

[0066] Optionally, the method further comprises:

[0067] Extracting features from at least a portion of the image frames in the acquired video data to obtain image features;

[0068] In the case where the video data includes audio data, extracting features from the audio data to obtain audio features;

[0069] The multimodal data also includes the image features and the audio features.

[0070] In this embodiment, while performing feature extraction on the acquired video data, feature extraction can also be performed on at least part of the image frames in the acquired video data. For example, image frames are extracted from the video at a certain frame rate interval (for example, one frame is extracted every 1 / 5 second), and feature extraction is performed on at least part of the image frames in the video data to reduce data redundancy while retaining important video information.

[0071] And when the video data includes audio data, the audio data can be first processed by dividing the frames, and the duration of each frame is set according to the characteristics of the audio and the analysis requirements (such as 20 milliseconds per frame). For each frame of audio, audio features are extracted, such as Mel frequency cepstral coefficients, which can effectively characterize the spectral characteristics of the audio and reflect the timbre, pitch and other information of the audio. The extracted audio features are normalized to keep them consistent in the numerical range, so as to facilitate fusion with other modal features, and finally obtain the audio feature vector.

[0072] Specifically, Figure 4As shown, the data in the multimodal content input module of the input server may include video data, image data, audio data and text data. Among them, text processing includes two parts: word tokenization and text encoding, and the main purpose is to convert text data into a vector form. For the processing of video data, image data and audio data, most methods in the related art encode images and videos into independent feature spaces, and then input them as input data into a large language model. However, due to the lack of unified tokenization of images and videos, it becomes very challenging for large language models to learn multimodal interactions from several poor projection layers. Therefore, the visual representation is unified into the language feature space, and the representation is aligned to a unified visual feature space. Then, a multimodal joint learning method is used for feature fusion. Since the visual representation is aligned before projection, a shared projection layer is used to map a unified visual representation for the large model. At the same time, in order to improve computational efficiency, images and videos are jointly trained to improve the ability to understand images and videos. The unified visual representation is conducive to the large model learning to process images and videos at the same time, verifying the complementarity of the modalities, and the effect is better than that of models designed specifically for images or videos. Finally, the decoding module is used to decode the multimodal joint information into text. This process uses vectorized multimodal fusion information and a generative method to predict the next word to convert the multimodal information learned by the model into coherent continuous text. In this way, the powerful learning and understanding capabilities of the multimodal model are utilized to achieve accurate understanding of multimodal video content and high-quality information generation.

[0073] Optionally, the extracting features from the acquired video data to obtain video features includes:

[0074] Performing frame rate conversion processing and size conversion processing on the video data to obtain a plurality of image frames, wherein intervals between adjacent frames in the plurality of image frames are the same, and the intervals match a frame rate corresponding to a first preset video type, and resolutions of the plurality of image frames are the same, and the resolutions match a resolution corresponding to the first preset video type, and the first preset video type is a video type determined according to key frames in the video data;

[0075] The multiple image frames are input into a video encoder to obtain the video features.

[0076] In this embodiment, through frame rate conversion, size conversion and the processing flow of the video encoder, video features containing rich static and dynamic information can be effectively extracted from the video data, laying a solid foundation for subsequent multimodal information fusion and accurate video content information generation. Compared with traditional video feature extraction methods, it has significant advantages in the accuracy and completeness of feature representation and adaptability to subsequent processing links.

[0077] Specifically, the first preset video type is determined according to the key frame in the video data, and then the frame rate matching it is selected for conversion processing. This is based on the characteristics of different types of video content. For example, for action videos (such as sports events, action movies, etc.), a higher frame rate (such as 60fps or higher) can capture the details of fast movements more clearly, avoid motion blur, and enable subsequent feature extraction to more accurately reflect the motion state of the object; while for some videos with relatively static scenes and slow motion changes (such as landscape documentaries, lecture videos, etc.), a lower frame rate (such as 25fps or 30fps) can not only meet the basic visual coherence requirements, but also reduce the amount of data and improve processing efficiency. Among them, in the frame rate conversion process, if the frame rate needs to be reduced, a sampling algorithm is often used, such as extracting frames from a high frame rate video at a certain interval, but simple sampling may cause picture jitter or information loss. To solve this problem, a frame rate conversion algorithm based on motion compensation can be used. The algorithm estimates and reconstructs the unextracted frames by analyzing the motion vectors between adjacent frames, thereby maintaining the coherence and smoothness of the picture while reducing the frame rate. On the contrary, if you want to increase the frame rate, you use an interpolation algorithm. For example, the common linear interpolation algorithm generates intermediate frames by linearly interpolating the pixel values ​​of adjacent frames in the time dimension. The more advanced super-resolution frame rate improvement algorithm based on deep learning uses neural networks to learn the spatiotemporal characteristics of the video and can generate more realistic and higher-quality intermediate frames.

[0078] Similarly, based on the first preset video type, a suitable resolution is determined for resizing. Factors to be considered include display device compatibility, storage space limitations, and computing resource requirements for subsequent processing. For example, if the video is mainly used for playback on mobile devices, a size that matches the common mobile phone screen resolution (such as 720x1280, 1080x2340, etc.) should be selected to reduce data transmission and storage requirements while ensuring smooth playback on mobile devices; for professional videos that require detailed analysis and processing (such as medical imaging videos, high-definition surveillance videos, etc.), it may be necessary to maintain a higher resolution, but it should also be reasonably adjusted according to the performance of the processing equipment to avoid slow processing speed due to excessive size. Among them, the scaling algorithms for resizing processing may include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. The nearest neighbor interpolation algorithm is simple and fast, but it may cause jagged edges on the image. The bilinear interpolation algorithm considers the values ​​of the four nearest neighbor pixels around the target pixel and performs linear interpolation in two directions to generate a smoother image, but it will make the image slightly blurred. The bicubic interpolation algorithm uses the weighted average value of the 16 nearest neighbor pixels around the target pixel to provide better image quality, but the computational cost is high. In practical applications, it is necessary to select a suitable algorithm based on the requirements for image quality and processing speed. At the same time, in order to avoid image deformation, the aspect ratio of the video must be maintained when resizing. The size of the other dimension can be calculated based on the aspect ratio of the original video and one dimension of the target size (such as width or height).

[0079] The encoder can be composed of multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layer uses convolution kernels to perform convolution operations with image frames to extract local features of image frames. Convolution kernels of different sizes and numbers can detect feature information at different levels, such as small convolution kernels for capturing detailed features and large convolution kernels for perceiving more macroscopic feature patterns. Pooling layers (such as maximum pooling or average pooling) reduce the resolution of feature maps through downsampling operations, reduce the amount of calculation and increase the robustness of features while retaining the main feature information. The fully connected layer is used to integrate and map the features extracted by the previous layers, converting them into vector representations suitable for subsequent processing, and finally obtaining vectors that can comprehensively represent video features.

[0080] Optionally, determining a target prompt in a preset database according to the acquired text data may include:

[0081] Vectorizing the text data to obtain text vector information;

[0082] The target prompt is determined in the preset database according to the text vector information.

[0083] In this embodiment, Figure 5As shown, considering that the expression method of the query text directly affects the retrieval results, a slight text change may result in a very different result. Directly using the original query text for retrieval may be simple and effective in many cases, but sometimes it may be necessary to make some changes to the query text to obtain better retrieval results. Based on this, through the vectorized retrieval method, the user query retrieves the relevant text block, and provides a prompt template for generating a reply, which improves the accuracy and pertinence of the entire system in processing video content information. Specifically, multiple prompts are stored in the preset database, and each prompt also needs to be vectorized in advance (using the same or compatible vectorization method as the text data) in order to compare and match with the input text vector. By calculating the similarity between the text vector and the prompt vector in the database, the target prompt that best matches the text data is determined, thereby providing strong support for video content understanding and information generation based on the multimodal model, and improving the accuracy and pertinence of the entire system in processing video content information.

[0084] Optionally, the multimodal data is mapped to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, including:

[0085] Calculating the correlation weights between each modal feature and other modal features in the multimodal data to dynamically allocate attention resources to the multimodal data;

[0086] A sum operation is performed on the multimodal data according to the attention resource to obtain video content information, wherein the video content information includes a fused multimodal feature vector.

[0087] In this embodiment, attention allocation is achieved by calculating the correlation weights between different modal features (such as video features and target prompt features). For each modal feature, the model evaluates its degree of association with other modal features. The higher the degree of association, the greater the attention weight assigned to the modal feature. For example, in a video content understanding scenario, if a person is shown pointing at an object in the video, and the text prompt mentions the name and function of the object, the model will calculate the high correlation weights between the character action features in the video and the object-related features in the text, thereby allocating more attention resources to these related features and ignoring irrelevant background information.

[0088] After calculating the correlation weights between the features of each modality and allocating attention resources, the summation operation is performed to fuse the multimodal features according to the attention weights. This allows the model to comprehensively consider the importance of information from different modalities, focus on the most critical features for understanding video content and generating information, and generate a fusion vector that contains multimodal key information and effectively enhances the semantic association between each modal information. For example, Figure 6 As shown in the figure, for a motion scene shown in a video, if the text prompt focuses on the characteristics of the food (for example, soft or salty), then in the summation operation, the video features related to the food (such as color features, texture features, and shape features, etc.) and text features (such as keyword vectors describing food attributes) will be weighted summed according to higher relevance weights, and the resulting fusion vector will highlight these key information, thereby providing strong support for the subsequent generation of accurate video content information (such as descriptions, comments, or questions and answers about food, etc.).

[0089] Among them, multimodal thinking chain reasoning can be used to enhance the reasoning accuracy of the multimodal model through two-stage reasoning. Specifically, the language (text data) and visual (video / image data) modalities are merged into a two-stage framework. The multimodal thinking chain includes two training stages: basic principle generation and answer inference. The two stages share the same model architecture but differ in input and output. In the basic principle generation stage, language input and visual input can be provided to the model to learn a basic principle generation model. In the answer reasoning stage, the text generated by the generation model is appended to the original language input to construct the language input in the second stage. Then, the updated language and video input are fed to the answer reasoning model to infer the final answer, and the model with the same architecture is used to achieve the purpose of improving the output effect.

[0090] Optionally, before mapping the multimodal data to a shared semantic embedding space based on the pre-trained multimodal model to generate video content information, the method further includes:

[0091] Acquire training data, wherein the training data includes a sample video set, a sample text set, a sample audio set, and corresponding information annotations;

[0092] Obtaining first model parameters corresponding to the first model after training based on the training data, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, the preset initial model includes a plurality of neural network layers, and the preset initial model is used to learn and process the fused multimodal features, and i is a positive integer;

[0093] The model parameters of the first model are updated based on the first model parameters to obtain the multimodal model, wherein the multimodal model is a model obtained by updating the preset initial model at the i+1th iteration.

[0094] In this embodiment, training data is first obtained to construct a large-scale data set containing rich and diverse multimodal data such as video, text, and audio. Video data covers different scenes (such as daily life, film and television clips, surveillance videos, teaching videos, etc.), different themes (such as entertainment, technology, culture, sports, etc.) and different types (such as animation, real-life shooting, documentaries, etc.), ensuring that the model can learn various visual information. Text data includes descriptions, comments, subtitles, questions, etc. related to the video, providing language-level explanations and supplements for the video content. Audio data (if applicable) collects sounds synchronized with the video, such as character dialogues, background music, environmental sound effects, etc., to enrich the model's understanding of audio modality. The collected data can be cleaned and preprocessed to remove noise data (such as blurred videos, garbled text, damaged audio), duplicate data, and irrelevant data to ensure the quality and validity of the data. At the same time, the video, text, and audio are annotated, and the annotation information can include object categories, character behaviors, scene types, semantic tags of text, event types of audio, etc. in the video, and these annotations will serve as supervision information for the training model.

[0095] In one example, recording data including communication between drivers and passengers can be collected from various vehicle environments, with particular attention paid to scenarios where sexually suggestive words or dirty words may appear, such as uncivilized words produced by drivers' irritability during traffic congestion, or inappropriate conversations between passengers. These data sources can be collected by recording equipment in actual operating vehicles, or by audio recorded by simulating relevant scenarios. Alternatively, for service scenarios such as taxis and online car-hailing, negative service conversations that may exist between drivers and passengers are collected, such as quarrels or dissatisfied exchanges caused by drivers' refusal to pick up passengers, bad attitudes, detours, etc., as well as unreasonable requests or disrespectful words from passengers to drivers. Data collection methods include cooperating with relevant operating platforms to obtain real case recordings, or recording and sampling service scenarios with permission. Then, the collected vehicle recording data is pre-processed, and first the audio format is uniformly converted to ensure that all audio data can be compatible with the subsequent processing system. Then, through audio signal processing technology, such as noise reduction processing, background noise (such as vehicle driving noise, wind noise, etc.) is removed to improve the clarity of the voice signal, so as to more accurately detect sexual innuendo, dirty words and negative service conversation content. At the same time, the audio is framed and the continuous audio stream is divided into shorter audio frames. The length of each frame is set according to the requirements of speech analysis (such as 20-50 milliseconds) to prepare for subsequent feature extraction.

[0096] In another example, surveillance cameras can be installed on different types of vehicles (such as buses, taxis, private cars, etc.) to collect video data of normal passenger transportation and possible illegal passenger transportation (such as overloading, illegal operation of non-operating vehicles, etc.). The data collection scope covers different time periods (daytime, nighttime), different road conditions (urban roads, highways, rural roads, etc.) and different regions (city centers, suburbs, remote areas, etc.) to ensure data diversity. Similarly, the on-board surveillance camera is used to focus on collecting video scenes of fights and conflicts between passengers in the car or between passengers and drivers. These data can come from actual public security incident records, or they can be recorded by simulating similar conflict scenes in a safe and controllable environment, and relevant normal riding scene videos are collected as comparison data. Then, the collected on-board surveillance video data is first converted into a video format and adjusted in resolution to meet the processing requirements. Since the on-board surveillance video may be affected by factors such as light changes (such as entering and exiting tunnels, under different weather conditions) and vehicle shaking, image stabilization processing is required to eliminate the picture jitter caused by vehicle movement and ensure that the movement trajectory of objects and people in the video can be accurately analyzed. Then, like the video processing flow mentioned above, frame rate conversion and size conversion are performed to obtain appropriate image frames, and then video features are extracted, including static features (such as color, shape, texture and other features of people and objects in the car, which are used to identify people's identities, object types, etc.) and dynamic features (such as people's movement trajectory, speed, vehicle driving status, etc., which are used to judge the number of people and whether there is a fight).

[0097] During the training process, an iterative training method is adopted to divide the data set into multiple batches. Each time, a batch of data is randomly selected from the data set and input into the model for forward propagation to calculate the prediction results. Then, the loss value is calculated according to the loss function, and the gradient of the loss to the model parameters is calculated through the back propagation algorithm. Finally, the optimization algorithm is used to update the parameters. The first model parameters corresponding to the first model after training based on the training data are obtained, and the model parameters of the first model are updated based on the first model parameters to obtain a multimodal model. In this way, a multimodal model that can effectively process multimodal data, map different modal information to a shared semantic embedding space, and accurately generate video content information can be gradually trained. In practical applications, the training process can be further optimized and adjusted according to specific task requirements, data characteristics, computing resources and other factors.

[0098] Further references Figure 7 , as a response to the above Figure 7 In order to realize the method shown in the figure, the present application provides an embodiment of an information generation device 700 based on a multimodal model, and the device embodiment is similar to Figure 7 Corresponding to the method embodiment shown, the device can be specifically applied in a server.

[0099] An embodiment of the present application provides an information generation device based on a multimodal model. The information generation device 700 based on a multimodal model includes:

[0100] A first feature extraction module 710 is used to extract features from the acquired video data to obtain video features, wherein the video features include static features and dynamic features, wherein the static features include color features, texture features, and shape features, and the dynamic features include camera parameter features, and speed vector features and position vector features of a target object in the video data;

[0101] A determination module 720, configured to determine a target prompt in a preset database according to the acquired text data, wherein the preset database stores a plurality of prompts, and the target prompt is a prompt in the preset database that matches the text data;

[0102] The generation module 730 is used to map the multimodal data to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, wherein the multimodal data includes the video features and the target prompts, and the video content information is information that describes the video data in the form of dialogue text.

[0103] Optionally, the device further comprises:

[0104] A second feature extraction module, used to extract features from at least part of the image frames in the acquired video data to obtain image features;

[0105] A third feature extraction module is used to extract features from the audio data to obtain audio features when the video data includes audio data;

[0106] The multimodal data also includes the image features and the audio features.

[0107] Optionally, the first feature extraction module 710 is specifically configured to:

[0108] Performing frame rate conversion processing and size conversion processing on the video data to obtain a plurality of image frames, wherein intervals between adjacent frames in the plurality of image frames are the same, and the intervals match a frame rate corresponding to a first preset video type, and resolutions of the plurality of image frames are the same, and the resolutions match a resolution corresponding to the first preset video type, and the first preset video type is a video type determined according to key frames in the video data;

[0109] The multiple image frames are input into a video encoder to obtain the video features.

[0110] Optionally, the determination module 720 is specifically configured to:

[0111] Vectorizing the text data to obtain text vector information;

[0112] The target prompt is determined in the preset database according to the text vector information.

[0113] Optionally, the device further comprises:

[0114] A first acquisition module is used to acquire training data, wherein the training data includes a sample video set, a sample text set, a sample audio set, and corresponding information annotations;

[0115] A second acquisition module is used to obtain first model parameters corresponding to the first model after training based on the training data, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, the preset initial model includes a plurality of neural network layers, and the preset initial model is used to learn and process the fused multimodal features, and i is a positive integer;

[0116] An updating module is used to update the model parameters of the first model based on the first model parameters to obtain the multimodal model, wherein the multimodal model is a model obtained by updating the preset initial model at the i+1th iteration.

[0117] Optionally, the generating module 730 is specifically configured to:

[0118] Calculating the correlation weights between each modal feature and other modal features in the multimodal data to dynamically allocate attention resources to the multimodal data;

[0119] A sum operation is performed on the multimodal data according to the attention resource to obtain video content information, wherein the video content information includes a fused multimodal feature vector.

[0120] To solve the above technical problems, the present application also provides an electronic device. Figure 8 , Figure 8 This is a basic structural block diagram of the electronic device of this embodiment.

[0121] The electronic device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected and communicated through a system bus. It should be noted that the figure only shows an electronic device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0122] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0123] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 8. Of course, the memory 81 can also include both the internal storage unit of the electronic device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the electronic device 8, such as the program code of the information generation method based on the multimodal model, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0124] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the electronic device 8. In this embodiment, the processor 82 is used to run the program code stored in the memory 81 or process data, such as running the program code of the information generation method based on the multimodal model.

[0125] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the electronic device 8 and other electronic devices.

[0126] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a program of the information generation method based on the multimodal model, and the program of the information generation method based on the multimodal model can be executed by at least one processor so that the at least one processor performs the steps of the information generation method based on the multimodal model as described above.

[0127] The present application also provides another implementation, namely, providing a computer program product, wherein the computer program product includes computer instructions, and when the computer instructions are executed by a processor, the steps of the above-mentioned multimodal model-based information generation method are implemented.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware online platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0129] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0130] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating information based on a multimodal model, characterized in that: The method comprises: Extracting features from the acquired video data to obtain video features, wherein the video features include static features and dynamic features; According to the acquired text data, a target prompt word may be determined in a preset database, wherein a plurality of prompt words are stored in the preset database, and the target prompt word is a prompt word in the preset database that matches the text data; Based on a pre-trained multimodal model, multimodal data are mapped to a shared semantic embedding space to generate video content information, wherein the multimodal data includes the video features and the target prompts, and the video content information is information that describes the video data in the form of dialogue text.

2. The method according to claim 1, characterized in that The method further comprises: Extracting features from at least a portion of the image frames in the acquired video data to obtain image features; In the case where the video data includes audio data, extracting features from the audio data to obtain audio features; The multimodal data also includes the image features and the audio features.

3. The method according to claim 1, characterized in that The step of extracting features from the acquired video data to obtain video features includes: Performing frame rate conversion processing and size conversion processing on the video data to obtain a plurality of image frames, wherein intervals between adjacent frames in the plurality of image frames are the same, and the intervals match a frame rate corresponding to a first preset video type, and resolutions of the plurality of image frames are the same, and the resolutions match a resolution corresponding to the first preset video type, and the first preset video type is a video type determined according to key frames in the video data; The multiple image frames are input into a video encoder to obtain the video features.

4. The method according to claim 1, characterized in that The step of determining a target prompt in a preset database according to the acquired text data may include: Vectorizing the text data to obtain text vector information; The target prompt is determined in the preset database according to the text vector information.

5. The method according to claim 1, characterized in that Before mapping the multimodal data to a shared semantic embedding space based on the pre-trained multimodal model to generate video content information, the method further includes: Acquire training data, wherein the training data includes a sample video set, a sample text set, a sample audio set, and corresponding information annotations; Obtaining first model parameters corresponding to the first model after training based on the training data, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, the preset initial model includes a plurality of neural network layers, and the preset initial model is used to learn and process the fused multimodal features, and i is a positive integer; The model parameters of the first model are updated based on the first model parameters to obtain the multimodal model, wherein the multimodal model is a model obtained by updating the preset initial model at the i+1th iteration.

6. The method according to claim 1, characterized in that The multimodal model based on pre-training maps the multimodal data to a shared semantic embedding space to generate video content information, including: Calculating the correlation weights between each modal feature and other modal features in the multimodal data to dynamically allocate attention resources to the multimodal data; A sum operation is performed on the multimodal data according to the attention resource to obtain video content information, wherein the video content information includes a fused multimodal feature vector.

7. An information generation device based on a multimodal model, characterized in that: The device comprises: A first feature extraction module is used to extract features from the acquired video data to obtain video features, wherein the video features include static features and dynamic features; A determination module, configured to determine a target prompt in a preset database according to the acquired text data, wherein the preset database stores a plurality of prompts, and the target prompt is a prompt in the preset database that matches the text data; A generation module is used to map multimodal data to a shared semantic embedding space based on a pre-trained multimodal model to generate video content information, wherein the multimodal data includes the video features and the target prompts, and the video content information is information that describes the video data in the form of dialogue text.

8. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the information generation method based on the multimodal model as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating information based on a multimodal model as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the information generation method based on a multimodal model as described in any one of claims 1 to 6.

Images