Multi-round real-time multi-modal large model interaction method, related device and storage medium

Abstract

Embodiments of the present application relate to the field of artificial intelligence, and provide a multi-round real-time multi-modal large model interaction method, related devices and a storage medium. The multi-round real-time multi-modal large model interaction method comprises: converting a first target audio into text to obtain a first target task description text; determining a first key frame image set based on a first target video; performing image segmentation on each key frame image in the first key frame image set to obtain an image segmentation result of each key frame image in the first key frame image set, the image segmentation result comprising a plurality of image segmentation regions and corresponding region labels; processing the first target task description text and the image segmentation result of each key frame image in the first key frame image set based on a target multi-modal large model to obtain a first output text; and converting the first output text into speech to obtain a first output audio. The present application can improve the accuracy of multi-modal large model interaction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically to a multi-turn real-time multimodal large model interaction method, related apparatus and storage medium. Background Technology

[0002] Large Language Models (LLMs), with their superior natural language generation capabilities, are now able to generate grammatically correct, persuasive texts that closely resemble human writing. LLMs have been applied in numerous fields, bringing convenience and efficiency improvements and playing a crucial role in areas such as education, content creation, language translation, programming assistance, scientific research, information retrieval, entertainment, and games. However, the accuracy of current large model interactions remains relatively low. Summary of the Invention

[0003] This application provides a multi-turn real-time multimodal large model interaction method, related device, and storage medium, which can improve the accuracy of multi-turn real-time multimodal large model interaction.

[0004] In a first aspect, embodiments of this application provide a multi-round real-time multimodal large model interaction method, the method comprising:

[0005] Obtain the first target audio input by the user and the first target video corresponding to the first target audio;

[0006] The first target audio is converted into text to obtain the first target task description text;

[0007] A first set of keyframe images is determined based on the first target video, and the first set of keyframe images includes multiple keyframe images in the first target video.

[0008] Image segmentation is performed on each of the keyframe images in the first keyframe image set to obtain image segmentation results for each of the keyframe images in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0009] Based on the target multimodal large model, the image segmentation results of the first target task description text and each keyframe image in the first keyframe image set are processed to obtain the first output text;

[0010] The first output text is converted into speech to obtain the first output audio.

[0011] In one embodiment, the multi-round real-time multimodal large model interaction method further includes:

[0012] Acquire a second target audio input by the user and a second target video corresponding to the second target audio, wherein the input time of the second target audio is later than that of the first target audio;

[0013] The second target audio is converted into text to obtain the second target task description text;

[0014] A second set of keyframe images is determined based on the second target video, and the second set of keyframe images includes multiple keyframe images in the second target video;

[0015] Image segmentation is performed on each of the keyframe images in the second keyframe image set to obtain image segmentation results for each of the keyframe images in the second keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0016] Based on the target multimodal large model, the second target task description text, the image segmentation results of each keyframe image in the second keyframe image set, the first target task description text, the image segmentation results of each keyframe image in the first keyframe image set, and the first output text are processed to obtain the second output text;

[0017] The second output text is converted into speech to obtain the second output audio.

[0018] In one implementation, determining the first set of keyframe images based on the first target video includes:

[0019] The first target video is divided into multiple video segments;

[0020] Based on the video segments, a set of third keyframe images is determined, resulting in multiple sets of third keyframe images corresponding to multiple video segments;

[0021] The first keyframe image set is obtained by merging the multiple sets of third keyframe images corresponding to the multiple video segments.

[0022] In one implementation, determining the third keyframe image set based on the video segment includes:

[0023] Multiple video frames in the video segment are clustered to obtain multiple frame clusters;

[0024] The third keyframe image set is determined based on the central video frame image corresponding to the cluster center of each frame cluster.

[0025] In one implementation, determining the third keyframe image set based on the video segment includes:

[0026] Obtain motion change parameters of the video segment, wherein the larger the motion change parameters, the faster the motion change of the video segment;

[0027] The number of target images corresponding to the video segment is determined based on the motion change parameters of the video segment, wherein the larger the motion change parameters, the larger the number of target images.

[0028] The number of video frames representing the target image are extracted from the video segment and used as the third set of keyframe images.

[0029] In one implementation, the target multimodal large model includes multiple different encoding modules, modality fusion modules, and decoding modules. The image segmentation results based on the target multimodal large model, processing the first target task description text and each keyframe image in the first keyframe image set, yield a first output text, including:

[0030] The first target task description text and the image segmentation results of each keyframe image in the first keyframe image set are encoded using different encoding modules in the target multimodal large model to obtain a first feature vector corresponding to the first target task description text and a second feature vector corresponding to the image segmentation results of each keyframe image in the first keyframe image set.

[0031] The modality fusion module in the target multimodal large model is used to weight and fuse the first feature vector and the second feature vector to obtain a fused feature vector;

[0032] The fused feature vector is processed using the decoding module in the target multimodal large model to obtain the first output text.

[0033] Secondly, embodiments of this application provide a multi-turn real-time multimodal large model interaction device, which has the function of implementing the multi-turn real-time multimodal large model interaction method corresponding to the first aspect provided above. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, and the modules can be software and / or hardware.

[0034] In one embodiment, the multi-round real-time multimodal large model interaction device includes:

[0035] The acquisition module is configured to acquire the first target audio input by the user and the first target video corresponding to the first target audio;

[0036] The first conversion module is configured to convert the first target audio into text to obtain the first target task description text;

[0037] The determining module is configured to determine a first set of keyframe images based on the first target video, the first set of keyframe images including multiple keyframe images in the first target video;

[0038] The segmentation module is configured to perform image segmentation on each of the keyframe images in the first keyframe image set to obtain image segmentation results for each of the keyframe images in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0039] The processing module is configured to process the image segmentation results of the first target task description text and each of the keyframe images in the first keyframe image set based on the target multimodal large model to obtain the first output text;

[0040] The second conversion module is configured to convert the first output text into speech to obtain the first output audio.

[0041] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the multi-round real-time multimodal large model interaction method as described in the first aspect.

[0042] Fourthly, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-round real-time multimodal large model interaction method described in the first aspect.

[0043] Fifthly, embodiments of this application provide a chip that includes a processor coupled to a transceiver of a terminal device, for executing the technical solution provided in the first aspect of embodiments of this application.

[0044] In a sixth aspect, embodiments of this application provide a chip system including a processor for supporting a terminal device in implementing the functions involved in the first aspect above, such as generating or processing information involved in the image processing method provided in the first aspect above.

[0045] In one possible design, the aforementioned chip system also includes a memory for storing program instructions and data necessary for the terminal. The chip system can be composed of chips or may include chips and other discrete components.

[0046] In a seventh aspect, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a computer, cause the computer to execute the multi-round real-time multimodal large model interaction method provided in the first aspect.

[0047] Compared to existing technologies, in this application embodiment, the process involves: acquiring a first target audio input by the user and a first target video corresponding to the first target audio; converting the first target audio into text to obtain a first target task description text; determining a first set of keyframe images based on the first target video, the first set of keyframe images including multiple keyframe images in the first target video; performing image segmentation on each keyframe image in the first set of keyframe images to obtain image segmentation results for each keyframe image in the first set of keyframe images, the image segmentation results including multiple image segmentation regions and corresponding region labels; processing the first target task description text and the image segmentation results of each keyframe image in the first set of keyframe images based on a target multimodal large model to obtain a first output text; and converting the first output text into speech to obtain a first output audio. This application, after acquiring the first target audio and the first target video corresponding to the first target audio, converts the audio into text and extracts the keyframe images of the first target video for segmentation processing. This allows the use of the text converted from audio and the keyframe images to predict the output text and convert it back into audio, thereby improving the accuracy of multimodal large model interaction. Attached Figure Description

[0048] The objectives, features, and advantages of the embodiments of this application will become readily understood by referring to the accompanying drawings and the detailed description of the embodiments. Wherein:

[0049] Figure 1 This is a schematic diagram of a multi-round real-time multimodal large model interaction system, which is an embodiment of the multi-round real-time multimodal large model interaction method in this application.

[0050] Figure 2 This is a flowchart illustrating a multi-round real-time multimodal large model interaction method according to an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of information interaction for a multi-round real-time multimodal large model interaction method according to an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of a multi-round real-time multimodal large model interaction device according to an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of a computing device according to an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of the structure of a mobile phone in one embodiment of this application;

[0055] Figure 7 This is a schematic diagram of a server structure in one embodiment of this application.

[0056] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0057] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be omitted or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, indirect coupling between modules, or electrical or other similar forms of communication connection, none of which are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0058] The general-purpose big model is currently widely used in applications such as chat dialogue, text editing, artistic creation, coding, mathematical reasoning, and bioinformatics. Although it has created many new business models and has powerful capabilities, after the general-purpose big model was launched for users, there are mainly algorithmic risks, data risks, and application risks in the three types of applications: translation, chat, and collaboration.

[0059] This application also provides a multi-turn real-time multimodal large model interaction method, related apparatus, and storage medium, which can be applied to a multi-turn real-time multimodal large model interaction system. This system can include a multi-turn real-time multimodal large model interaction device, which can be deployed integratedly or separately. The device is used to at least acquire user input data and multiple first attack action types; randomly select one first attack action type from the multiple first attack action types as the target attack action type; select an attack action from the set of attack actions corresponding to the target attack action type as the target attack action; process the user input data based on the target attack action to obtain target input data; and attack the tested model based on the target input data to obtain the attack result.

[0060] The solutions provided in this application involve technologies such as Artificial Intelligence (AI) and Machine Learning (ML), and are specifically illustrated through the following embodiments:

[0061] AI, or Artificial Intelligence, refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, Artificial Intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial Intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0062] AI technology is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] The accuracy of existing large-scale model interactions is relatively low.

[0064] Compared to existing technologies, in this application embodiment, the process involves: acquiring a first target audio input by the user and a first target video corresponding to the first target audio; converting the first target audio into text to obtain a first target task description text; determining a first set of keyframe images based on the first target video, the first set of keyframe images including multiple keyframe images from the first target video; performing image segmentation on each keyframe image in the first set of keyframe images to obtain image segmentation results for each keyframe image in the first set of keyframe images, the image segmentation results including multiple image segmentation regions and corresponding region labels; processing the first target task description text and the image segmentation results of each keyframe image in the first set of keyframe images based on a target multimodal large model to obtain a first output text; and converting the first output text into speech to obtain a first output audio. This application, after acquiring the first target audio and the first target video corresponding to the first target audio, converts the audio into text and extracts the keyframe images from the first target video for segmentation processing. This allows the use of the text converted from audio and the keyframe images to predict the output text and convert it back into audio, thereby improving the accuracy of multimodal large model interaction.

[0065] In some implementations, reference is made to Figure 1The multi-round real-time multimodal large model interaction method provided in this application embodiment can be based on Figure 1 The diagram illustrates an implementation of a multi-turn real-time multimodal large model interaction system. This system may include an electronic device 100 and a memory 200. The electronic device 100 may be a server or a terminal device.

[0066] It should be noted that the server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0067] The terminal devices involved in the embodiments of this application can be devices that provide voice and / or data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. Examples include mobile phones (or "cellular" phones) and computers with mobile terminals, such as portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and / or data with a wireless access network. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and other devices.

[0068] Reference Figure 2 , Figure 2 This is a flowchart illustrating a multi-round real-time multimodal large model interaction method provided in an embodiment of this application. The method can be executed by a multi-round real-time multimodal large model interaction device. The method includes steps 101-106:

[0069] Step 101: Obtain the first target audio input by the user and the first target video corresponding to the first target audio.

[0070] In this application, the first target audio is the user's voice information recorded by a microphone or other recording device. The first target audio and the first target video are played synchronously.

[0071] Specifically, during the playback of the first target video, the user-inputted first target audio is acquired, resulting in the first target audio and its corresponding first target video. The recording start time and playback start time of the first target audio are the same, as are the recording end time and playback end time of the first target video.

[0072] In one specific embodiment, an input video stream is acquired. During the playback of the video stream, a first target audio input by the user is acquired, and a first target video corresponding to the first target audio is extracted from the video stream based on the first target audio, thereby obtaining the first target audio and the first target video corresponding to the first target audio.

[0073] Step 102: Convert the first target audio into text to obtain the first target task description text.

[0074] In this embodiment, the first target audio is input into a speech-to-text model and converted into text to obtain the first target task description text. The speech-to-text (STT) model is the core implementation of speech recognition, aiming to automatically convert audio signals into text sequences. This technology is fundamental to human-computer interaction, intelligent assistants, and speech translation, and its framework encompasses multiple fields such as signal processing, machine learning, and natural language understanding. The speech-to-text model can be a Kaldi model, a SpeechRecognition model, or a Transformer-based model, depending on the specific requirements.

[0075] Step 103: Determine a first set of keyframe images based on the first target video. The first set of keyframe images includes multiple keyframe images from the first target video.

[0076] In one specific embodiment, the color histogram difference between two adjacent video frames in the first target video is calculated. When the color histogram difference exceeds a preset color threshold, the two video frames are determined to be keyframe images. This method can effectively capture scenes with significant color changes in video. The preset color threshold can be set according to specific circumstances.

[0077] In another specific embodiment, the structural similarity index (SSIM) between two adjacent video frames in the first target video is calculated. Two video frames with a SSIM lower than a first preset similarity index threshold are identified as keyframes, and frames with rich structural information are extracted as keyframes. This method is more sensitive to structural changes in the video content. The first preset similarity index threshold can be set according to specific circumstances.

[0078] In another specific embodiment, the motion intensity between two adjacent video frames in the first target video is calculated. Two video frames with motion intensity exceeding a preset intensity threshold are identified as keyframes, and the frame with the larger motion amplitude is extracted as the keyframe. This method is suitable for capturing scenes with significant object movement in the video. The preset intensity threshold can be set according to specific circumstances.

[0079] In another specific embodiment, a keyframe deep learning model is constructed. The input is a sequence of video frames, and the output is the importance score for each frame. Frames with high importance scores are then selected as keyframes. The video frame sequence corresponding to the first target video is input into the keyframe deep learning model to obtain the importance score of each video frame in the first target video. Video frame images with importance scores higher than a preset score are determined as keyframe images. The preset score can be set according to specific circumstances.

[0080] In this embodiment of the application, determining the first keyframe image set based on the first target video includes:

[0081] (1) Divide the first target video into multiple video segments.

[0082] In this embodiment, a sliding window of preset length is used to slide on the first target video at preset steps to obtain multiple video segments of preset length.

[0083] (2) Determine the set of third keyframe images based on video segments to obtain multiple sets of third keyframe images corresponding to multiple video segments.

[0084] In one specific embodiment of this application, the color histogram difference between two adjacent video frame images in a video segment is calculated. When the color histogram difference exceeds a preset color threshold, the two video frame images are determined to be keyframe images.

[0085] In another specific embodiment, the structural similarity index between two adjacent video frame images in a video segment is calculated, and two video frame images with a structural similarity index lower than a first preset similarity index threshold are identified as key frame images.

[0086] In another specific embodiment, the motion intensity between two adjacent video frame images in a video segment is calculated, and the two video frame images with motion intensity higher than a preset intensity threshold are determined as key frame images.

[0087] In another specific embodiment, a keyframe deep learning model is constructed. The input is a sequence of video frames, and the output is the importance score of each frame. Frames with high importance scores are then selected as keyframes. The video frame sequence corresponding to a video segment is input into the keyframe deep learning model to obtain the importance score of each video frame image in the video segment. Video frame images with importance scores higher than a preset score are determined as keyframe images.

[0088] In yet another specific embodiment, determining the third set of keyframe images based on video clips includes:

[0089] 1-1: Cluster multiple video frames in a video segment to obtain multiple frame clusters.

[0090] 1-2: Determine the set of third keyframe images based on the central video frame images corresponding to the cluster centers of each frame cluster.

[0091] In one specific embodiment, the central video frame image corresponding to the cluster center of multiple frame clusters is determined as the third keyframe image set.

[0092] In another specific embodiment, the structural similarity index between two adjacent video frames in a video segment is calculated. Two video frames with a structural similarity index lower than a first preset similarity index threshold are identified as keyframe images and placed into a fifth keyframe image set. The central video frame images corresponding to the cluster centers of multiple frame clusters are identified as a fourth keyframe image set. The structural similarity index between each central video frame image in the fourth keyframe image set and each keyframe image in the fifth keyframe image set is calculated. The maximum value of the structural similarity index between each keyframe image and the central video frame image is identified as the maximum similarity index corresponding to the central video frame image. The central video frame images with the highest number of target images ranked from largest to smallest maximum similarity index are identified as a third keyframe image set.

[0093] In another specific embodiment, the motion intensity between two adjacent video frame images in a video segment is calculated, and the two video frame images with motion intensity higher than a preset intensity threshold are identified as key frame images and placed into the fifth key frame image set.

[0094] In another specific embodiment, a keyframe deep learning model is constructed. The input is a sequence of video frames, and the output is the importance score of each frame. Then, frames with high importance scores are selected as keyframes. The sequence of video frames corresponding to a video segment is input into the keyframe deep learning model to obtain the importance score of each video frame image in the video segment. Video frame images with importance scores higher than a preset score are determined as keyframe images and added to the fifth keyframe image set.

[0095] (3) Merge multiple sets of third keyframe images corresponding to multiple video segments to obtain a set of first keyframe images.

[0096] Furthermore, a third set of keyframe images is determined based on the video clips, including:

[0097] (1) Obtain the motion change parameters of the video clip. The larger the motion change parameters, the faster the motion change of the video clip.

[0098] In one specific embodiment, the motion intensity between two adjacent video frames in a video segment is calculated to obtain multiple motion intensities in the video segment, and the average value of the multiple motion intensities is determined as the motion change parameter of the video segment.

[0099] (2) Determine the number of target images corresponding to a video segment based on the motion change parameters of the video segment.

[0100] The greater the motion change parameter, the larger the number of target images. This application dynamically adjusts the keyframe extraction threshold based on the characteristics of the video content. For example, for videos with rapid motion changes, the threshold is lowered to extract more keyframes.

[0101] (3) Extract the number of video frame images of the target image from the video clip and use them as the third keyframe image set.

[0102] In one specific embodiment, multiple video frames in a video segment are clustered to obtain frame clusters representing the target number of images. A third set of keyframe images is determined based on the center video frame image corresponding to the cluster center of each frame cluster. For example, the number of target images is determined as the number of clusters K, and K-means clustering is used to cluster multiple video frames in the video segment to obtain frame clusters representing the target number of images. Specifically, K cluster centers are randomly initialized; each video frame is assigned to the nearest center, forming K clusters; the mean of each cluster is recalculated as the new center, and this process is iterated until the centers no longer change significantly or the maximum number of iterations is reached.

[0103] In another specific embodiment, multiple video frames in a video segment are clustered based on a preset number of images to obtain a frame cluster with a preset number of images, where the preset number of images is greater than the target number of images. The target number of center video frame images are then obtained from the center video frame images of the preset number of images to obtain a third set of keyframe images.

[0104] Specifically, with a preset number of images of N, multiple video frames are randomly and equally divided into Q video frame sets. Each of the Q video frame sets is then clustered into N second video frame clusters. Specifically, the K-means algorithm is used to cluster the Q video frame sets, resulting in N second video frame clusters for each set. Each second video frame cluster is designated as the target video frame cluster, and the similarity between the target video frame cluster and the second video frame clusters in the Q video frame sets is calculated. The second video frame clusters with the highest similarity to the target video frame cluster are merged to obtain the first video frame cluster corresponding to the target video frame cluster. This results in N first video frame clusters corresponding to the N second video frame clusters. Each first video frame cluster is then designated as a frame cluster, resulting in N frame clusters corresponding to the N first video frame clusters.

[0105] Specifically, the structural similarity index between each central video frame image in the fourth keyframe image set and each keyframe image in the fifth keyframe image set is calculated. Two video frame images with a structural similarity index lower than a first preset similarity index threshold are identified as keyframe images and added to the fifth keyframe image set. The central video frame images corresponding to the cluster centers of multiple frame clusters are identified as the fourth keyframe image set. The structural similarity index between each central video frame image in the fourth keyframe image set and each keyframe image in the fifth keyframe image set is calculated. The maximum value of the structural similarity index between each keyframe image and the central video frame image is identified as the maximum similarity index corresponding to the central video frame image. The central video frame images with the highest number of target images in descending order of maximum similarity index are identified as the third keyframe image set.

[0106] Furthermore, multiple threads are initiated to synchronously determine multiple sets of third keyframe images corresponding to multiple video segments. Multithreading technology is used to process video frames in parallel, improving the speed of keyframe extraction.

[0107] Step 104: Perform image segmentation on each keyframe image in the first keyframe image set to obtain the image segmentation results of each keyframe image in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0108] In this embodiment of the application, each keyframe image in the first keyframe image set is input into an image segmentation model for image segmentation to obtain the image segmentation result of each keyframe image in the first keyframe image set.

[0109] Image segmentation is the process of dividing an image into multiple meaningful regions. Based on segmentation granularity and task objectives, it can be divided into: Semantic Segmentation: classifies each pixel into a predefined category (e.g., "person," "vehicle," "background"), without distinguishing between different instances of the same category. Instance Segmentation: distinguishes not only pixel categories but also different instances of the same category (e.g., different people, different vehicles). Panoptic Segmentation: combines semantic and instance segmentation, processing both semantic categories (e.g., sky, road) and instance objects (e.g., people, vehicles).

[0110] Image segmentation models can perform entity segmentation and entity recognition, resulting in multiple segmented image regions and corresponding region labels. Each segmented image region is the area containing a different entity (such as a person, object, or background), and each region label is a label for that segmented region; for example, the region label might be "person," "object," or "background." The image segmentation model can be Mask R-CNN, DeepLab, etc., depending on the specific requirements.

[0111] Step 105: Based on the target multimodal large model, process the image segmentation results of the first target task description text and each keyframe image in the first keyframe image set to obtain the first output text.

[0112] In this embodiment, the target multimodal large model includes multiple different encoding modules, modality fusion modules, and decoding modules. Based on the target multimodal large model, the image segmentation results of the first target task description text and each keyframe image in the first keyframe image set are processed to obtain the first output text, including:

[0113] (1) Use different encoding modules in the target multimodal large model to encode the image segmentation results of the first target task description text and each key frame image in the first key frame image set, respectively, to obtain the first feature vector corresponding to the first target task description text and the second feature vector corresponding to the image segmentation results of each key frame image in the first key frame image set.

[0114] In this embodiment, different encoding modules in the target multimodal large model can be CNN, Transformer, etc. Keyframe images, segmentation maps and labels, textual descriptions of the task, and historical data are encoded separately to obtain their feature vector representations.

[0115] (2) Use the modal fusion module in the target multimodal large model to weight and fuse the first feature vector and the second feature vector to obtain the fused feature vector.

[0116] Specifically, the modality fusion module in the target multimodal large model can perform weighted fusion of the first feature vector and the second feature vector through early fusion, mid-term fusion, or late-term fusion.

[0117] Early fusion: Fusion is performed before the input to the large model, concatenating the first and second feature vectors together. Mid-stage fusion: Fusion is performed in the middle layers of the large model, using an attention mechanism to weight and fuse the first and second feature vectors. Late-stage fusion: Fusion is performed at the output layer of the large model, obtaining the speech and video outputs separately, and then performing a weighted sum.

[0118] (3) Use the decoding module in the target multimodal large model to process the fused feature vectors and obtain the first output text.

[0119] Based on the fused feature vector, the first output text, i.e., the textual description of the task result, is generated. Both autoregressive and non-autoregressive decoders can be used.

[0120] Furthermore, in this embodiment, the multi-round real-time multimodal large model interaction method includes: acquiring audio samples, corresponding video samples, and labeled output text; converting the audio samples into text to obtain task description text samples; determining a keyframe image sample set based on the video samples, performing image segmentation on each keyframe image in the keyframe image sample set to obtain image segmentation results for each keyframe image in the keyframe image sample set, the image segmentation results including multiple image segmentation regions and corresponding region labels; inputting the task description text samples and the image segmentation results of each keyframe image in the keyframe image sample set into the target multimodal large model to obtain predicted output text; calculating cross-entropy loss, multi-task loss, and contrastive loss based on the predicted output text and labeled output text, determining the total loss based on the cross-entropy loss, multi-task loss, and contrastive loss, and iteratively training the target multimodal large model based on the total loss until the total loss is less than a preset loss value. Cross-entropy loss: used to generate text descriptions of task results. Multi-task loss: used when multiple tasks need to be completed simultaneously (such as object detection, image description, etc.). Contrastive loss: used to learn the correspondence between different modal data.

[0121] Step 106: Convert the first output text into speech to obtain the first output audio.

[0122] In this embodiment, a text-to-speech (TTS) model is used to convert the first output text into speech, resulting in the first output audio. The first output audio is then played.

[0123] Furthermore, multi-round real-time multimodal large model interaction methods also include:

[0124] (1) Obtain the second target audio input by the user and the second target video corresponding to the second target audio, wherein the input time of the second target audio is later than that of the first target audio.

[0125] Specifically, the process involves acquiring the input video stream, acquiring the first target audio input by the user during video stream playback, and extracting the first target video corresponding to the first target audio from the video stream based on the first target audio, thus obtaining the first target audio and the first target video corresponding to the first target audio; then, acquiring the second target audio input by the user, and extracting the second target video corresponding to the second target audio from the video stream based on the second target audio, thus obtaining the second target audio and the second target video corresponding to the second target audio.

[0126] (2) Convert the audio of the second target into text to obtain the description text of the second target task.

[0127] (3) Determine a second set of keyframe images based on the second target video. The second set of keyframe images includes multiple keyframe images in the second target video.

[0128] (4) Perform image segmentation on each keyframe image in the second keyframe image set to obtain the image segmentation results of each keyframe image in the second keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0129] (5) Based on the target multimodal large model, process the image segmentation results of the second target task description text, the image segmentation results of each key frame image in the second key frame image set, the first target task description text, the image segmentation results of each key frame image in the first key frame image set, and the first output text to obtain the second output text.

[0130] Specifically, different encoding modules within the target multimodal large model are used to encode the second target task description text, the image segmentation results of each keyframe image in the second keyframe image set, the first target task description text, the image segmentation results of each keyframe image in the first keyframe image set, and the first output text, respectively. This yields the first feature vector corresponding to the second target task description text, the second feature vector corresponding to the image segmentation results of each keyframe image in the second keyframe image set, and the historical feature vector corresponding to the encoded first target task description text, the image segmentation results of each keyframe image in the first keyframe image set, and the first output text. The modality fusion module within the target multimodal large model then weights and fuses the first feature vector, the second feature vector, and the historical feature vector to obtain a fused feature vector. Finally, the decoding module within the target multimodal large model processes the fused feature vector to obtain the second output text.

[0131] (6) Convert the second output text into speech to obtain the second output audio.

[0132] This application implements multimodal large-scale model interaction, capable of processing and understanding various types of data, improving the naturalness and richness of the interaction. It supports multi-turn real-time interaction, better meeting users' complex interaction needs and improving the efficiency and practicality of the interaction. Through component-based orchestration, different functional modules can be flexibly combined, facilitating system expansion and customization. A keyframe extraction method based on speech correlation is proposed, improving the accuracy and efficiency of keyframe extraction. A processing method for multimodal data is proposed, effectively fusing various data types and improving the interactive performance of large models.

[0133] Improved keyframe extraction efficiency: Experimental data shows that compared with fixed frame extraction, invalid keyframes are reduced by 43%, and processing latency is reduced to 120ms / frame. Instruction relevance accuracy: In the COCO dataset test, the multimodal matching strategy achieves a relevance image selection accuracy of 92.7% (baseline 78.2%). Multi-turn dialogue consistency: The introduction of the memory network increases the task completion rate of 5 consecutive dialogues from 65% to 89%.

[0134] Furthermore, refer to Figure 3 , Figure 3 This is a flowchart illustrating a multi-round real-time multimodal large model interaction method provided in an embodiment of this application. The method can be executed by a multi-round real-time multimodal large model interaction device. The method includes the following steps:

[0135] (1) Obtain the first target audio input by the user and the first target video corresponding to the first target audio.

[0136] In one specific embodiment, an input video stream is acquired. During the playback of the video stream, a first target audio input by the user is acquired, and a first target video corresponding to the first target audio is extracted from the video stream based on the first target audio, thereby obtaining the first target audio and the first target video corresponding to the first target audio.

[0137] (2) Convert the audio of the first target into text to obtain the description text of the first target task.

[0138] (3) Determine a first set of keyframe images based on the first target video. The first set of keyframe images includes multiple keyframe images in the first target video.

[0139] (4) Perform image segmentation on each keyframe image in the first keyframe image set to obtain the image segmentation results of each keyframe image in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0140] (5) Based on the target multimodal large model, process the image segmentation results of the first target task description text and each key frame image in the first key frame image set to obtain the first output text.

[0141] (6) Convert the first output text into speech to obtain the first output audio.

[0142] (7) Obtain the second target audio input by the user and the second target video corresponding to the second target audio, wherein the input time of the second target audio is later than that of the first target audio.

[0143] (8) Convert the audio of the second target into text to obtain the description text of the second target task.

[0144] (9) Determine a second set of keyframe images based on the second target video. The second set of keyframe images includes multiple keyframe images in the second target video.

[0145] (10) Perform image segmentation on each keyframe image in the second keyframe image set to obtain the image segmentation results of each keyframe image in the second keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0146] (11) Based on the target multimodal large model, process the image segmentation results of the second target task description text, the image segmentation results of each key frame image in the second key frame image set, the first target task description text, the image segmentation results of each key frame image in the first key frame image set, and the first output text to obtain the second output text.

[0147] (12) Convert the second output text into speech to obtain the second output audio.

[0148] Compared to existing technologies, in this application embodiment, the process involves: acquiring a first target audio input by the user and a first target video corresponding to the first target audio; converting the first target audio into text to obtain a first target task description text; determining a first set of keyframe images based on the first target video, the first set of keyframe images including multiple keyframe images in the first target video; performing image segmentation on each keyframe image in the first set of keyframe images to obtain image segmentation results for each keyframe image in the first set of keyframe images, the image segmentation results including multiple image segmentation regions and corresponding region labels; processing the first target task description text and the image segmentation results of each keyframe image in the first set of keyframe images based on a target multimodal large model to obtain a first output text; and converting the first output text into speech to obtain a first output audio. This application, after acquiring the first target audio and the first target video corresponding to the first target audio, converts the audio into text and extracts the keyframe images of the first target video for segmentation processing. This allows the use of the text converted from audio and the keyframe images to predict the output text and convert it back into audio, thereby improving the accuracy of multimodal large model interaction.

[0149] See Figure 4 ,like Figure 4 The diagram shows a structural schematic of a multi-turn real-time multimodal large model interaction device. The multi-turn real-time multimodal large model interaction device in this embodiment can achieve the above-mentioned... Figure 2 The steps of the multi-round real-time multimodal large model interaction method executed in the corresponding embodiments are described above. The functions implemented by the multi-round real-time multimodal large model interaction device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The multi-round real-time multimodal large model interaction device 60 may include an acquisition module 601, a first conversion module 602, a determination module 603, a segmentation module 604, a processing module 605, and a second conversion module 606. The functional implementation of these modules can be found in [reference needed]. Figure 2 The operations performed in the corresponding embodiments will not be described in detail here.

[0150] The multi-round real-time multimodal large model interaction device includes:

[0151] The acquisition module 601 is configured to acquire a first target audio input by the user and a first target video corresponding to the first target audio.

[0152] The first conversion module 602 is configured to convert the first target audio into text to obtain the first target task description text;

[0153] The determining module 603 is configured to determine a first set of keyframe images based on the first target video, wherein the first set of keyframe images includes multiple keyframe images in the first target video.

[0154] The segmentation module 604 is configured to perform image segmentation on each of the keyframe images in the first keyframe image set to obtain image segmentation results for each of the keyframe images in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0155] Processing module 605 is configured to process the image segmentation results of the first target task description text and each of the keyframe images in the first keyframe image set based on the target multimodal large model to obtain the first output text;

[0156] The second conversion module 606 is configured to convert the first output text into speech to obtain the first output audio.

[0157] In one embodiment, the multi-round real-time multimodal large model interaction method further includes:

[0158] Acquire a second target audio input by the user and a second target video corresponding to the second target audio, wherein the input time of the second target audio is later than that of the first target audio;

[0159] The second target audio is converted into text to obtain the second target task description text;

[0160] A second set of keyframe images is determined based on the second target video, and the second set of keyframe images includes multiple keyframe images in the second target video;

[0161] Image segmentation is performed on each of the keyframe images in the second keyframe image set to obtain image segmentation results for each of the keyframe images in the second keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0162] Based on the target multimodal large model, the second target task description text, the image segmentation results of each keyframe image in the second keyframe image set, the first target task description text, the image segmentation results of each keyframe image in the first keyframe image set, and the first output text are processed to obtain the second output text;

[0163] The second output text is converted into speech to obtain the second output audio.

[0164] In one implementation, determining the first set of keyframe images based on the first target video includes:

[0165] The first target video is divided into multiple video segments;

[0166] Based on the video segments, a set of third keyframe images is determined, resulting in multiple sets of third keyframe images corresponding to multiple video segments;

[0167] The first keyframe image set is obtained by merging the multiple sets of third keyframe images corresponding to the multiple video segments.

[0168] In one implementation, determining the third keyframe image set based on the video segment includes:

[0169] Multiple video frames in the video segment are clustered to obtain multiple frame clusters;

[0170] The third keyframe image set is determined based on the central video frame image corresponding to the cluster center of each frame cluster.

[0171] In one implementation, determining the third keyframe image set based on the video segment includes:

[0172] Obtain motion change parameters of the video segment, wherein the larger the motion change parameters, the faster the motion change of the video segment;

[0173] The number of target images corresponding to the video segment is determined based on the motion change parameters of the video segment, wherein the larger the motion change parameters, the larger the number of target images.

[0174] The number of video frames representing the target image are extracted from the video segment and used as the third set of keyframe images.

[0175] In one implementation, the target multimodal large model includes multiple different encoding modules, modality fusion modules, and decoding modules. The image segmentation results based on the target multimodal large model, processing the first target task description text and each keyframe image in the first keyframe image set, yield a first output text, including:

[0176] The first target task description text and the image segmentation results of each keyframe image in the first keyframe image set are encoded using different encoding modules in the target multimodal large model to obtain a first feature vector corresponding to the first target task description text and a second feature vector corresponding to the image segmentation results of each keyframe image in the first keyframe image set.

[0177] The modality fusion module in the target multimodal large model is used to weight and fuse the first feature vector and the second feature vector to obtain a fused feature vector;

[0178] The fused feature vector is processed using the decoding module in the target multimodal large model to obtain the first output text.

[0179] The multi-turn real-time multimodal large model interaction device 60 in this application embodiment has been described above from the perspective of modular functional entities. The multi-turn real-time multimodal large model interaction device in this application embodiment is described below from the perspective of hardware processing.

[0180] Figure 4 The devices shown can all have the following characteristics: Figure 5 The structure shown, when Figure 4 The multi-turn real-time multimodal large model interaction device 60 shown has, for example, the following features: Figure 5 When the structure shown is used, Figure 5 The processor and transceiver in the device can perform the same or similar functions as the modules provided in the aforementioned device embodiments. Figure 5 The memory stores the computer programs that the processor needs to call when executing the above-mentioned multi-round real-time multimodal large model interaction method.

[0181] This application also provides a terminal device, such as... Figure 6 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal device can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0182] Figure 6 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 6 The mobile phone includes components such as a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will understand that... Figure 6 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0183] The following is combined with Figure 6 A detailed introduction to each component of a mobile phone:

[0184] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0185] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0186] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1080, and can also receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0187] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar display. Further, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 based on the type of touch event. Although in Figure 6 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0188] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0189] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.

[0190] Wi-Fi is a short-range wireless transmission technology. Through the Wi-Fi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 6 The Wi-Fi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0191] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020 and calls data stored in the memory 1020 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 1080.

[0192] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0193] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0194] In this embodiment of the application, the processor 1080 included in the mobile phone also has a process for controlling and executing the multi-round real-time multimodal large model interaction method executed by the multi-round real-time multimodal large model interaction device.

[0195] This application also provides a server; please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1122 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 may be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0196] Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0197] The steps performed by the server in the above embodiments can be based on this Figure 7 The structure of server 1100 shown. For example, as in the above embodiment, by Figure 7 The steps performed by the multi-round real-time multimodal large model interaction device 60 shown can be based on this Figure 7 The server architecture is shown. For example, the central processing unit 1122 performs the following operations by calling instructions from memory 1132:

[0198] Obtain the first target audio input by the user and the first target video corresponding to the first target audio;

[0199] The first target audio is converted into text to obtain the first target task description text;

[0200] A first set of keyframe images is determined based on the first target video, and the first set of keyframe images includes multiple keyframe images in the first target video.

[0201] Image segmentation is performed on each of the keyframe images in the first keyframe image set to obtain image segmentation results for each of the keyframe images in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0202] Based on the target multimodal large model, the image segmentation results of the first target task description text and each keyframe image in the first keyframe image set are processed to obtain the first output text;

[0203] The first output text is converted into speech to obtain the first output audio.

[0204] In one embodiment, the multi-round real-time multimodal large model interaction method further includes:

[0205] Acquire a second target audio input by the user and a second target video corresponding to the second target audio, wherein the input time of the second target audio is later than that of the first target audio;

[0206] The second target audio is converted into text to obtain the second target task description text;

[0207] A second set of keyframe images is determined based on the second target video, and the second set of keyframe images includes multiple keyframe images in the second target video;

[0208] Image segmentation is performed on each of the keyframe images in the second keyframe image set to obtain image segmentation results for each of the keyframe images in the second keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels.

[0209] Based on the target multimodal large model, the second target task description text, the image segmentation results of each keyframe image in the second keyframe image set, the first target task description text, the image segmentation results of each keyframe image in the first keyframe image set, and the first output text are processed to obtain the second output text;

[0210] The second output text is converted into speech to obtain the second output audio.

[0211] In one implementation, determining the first set of keyframe images based on the first target video includes:

[0212] The first target video is divided into multiple video segments;

[0213] Based on the video segments, a set of third keyframe images is determined, resulting in multiple sets of third keyframe images corresponding to multiple video segments;

[0214] The first keyframe image set is obtained by merging the multiple sets of third keyframe images corresponding to the multiple video segments.

[0215] In one implementation, determining the third keyframe image set based on the video segment includes:

[0216] Multiple video frames in the video segment are clustered to obtain multiple frame clusters;

[0217] The third keyframe image set is determined based on the central video frame image corresponding to the cluster center of each frame cluster.

[0218] In one implementation, determining the third keyframe image set based on the video segment includes:

[0219] Obtain motion change parameters of the video segment, wherein the larger the motion change parameters, the faster the motion change of the video segment;

[0220] The number of target images corresponding to the video segment is determined based on the motion change parameters of the video segment, wherein the larger the motion change parameters, the larger the number of target images.

[0221] The number of video frames representing the target image are extracted from the video segment and used as the third set of keyframe images.

[0222] In one implementation, the target multimodal large model includes multiple different encoding modules, modality fusion modules, and decoding modules. The image segmentation results based on the target multimodal large model, processing the first target task description text and each keyframe image in the first keyframe image set, yield a first output text, including:

[0223] The first target task description text and the image segmentation results of each keyframe image in the first keyframe image set are encoded using different encoding modules in the target multimodal large model to obtain a first feature vector corresponding to the first target task description text and a second feature vector corresponding to the image segmentation results of each keyframe image in the first keyframe image set.

[0224] The modality fusion module in the target multimodal large model is used to weight and fuse the first feature vector and the second feature vector to obtain a fused feature vector;

[0225] The fused feature vector is processed using the decoding module in the target multimodal large model to obtain the first output text.

[0226] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0227] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0228] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or modules, and may be electrical, mechanical, or other forms.

[0229] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0230] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0231] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0232] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0233] A computer program product includes one or more computer instructions. When a computer program is loaded and executed on a computer, it produces, in whole or in part, the flow or function according to the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0234] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A multi-turn real-time multimodal large model interaction method, characterized in that, The multi-round real-time multimodal large model interaction method includes: Obtain the first target audio input by the user and the first target video corresponding to the first target audio; The first target audio is converted into text to obtain the first target task description text; A first set of keyframe images is determined based on the first target video, the first set of keyframe images including multiple keyframe images in the first target video, wherein the first target video is divided into multiple video segments; a third set of keyframe images is determined based on the video segments, resulting in multiple sets of third keyframe images corresponding to multiple video segments, and the multiple sets of third keyframe images corresponding to multiple video segments are merged to obtain the first set of keyframe images; multiple video frames in the video segments are clustered to obtain multiple frame clusters; the third set of keyframe images is determined based on the center video frame image corresponding to the cluster center of each frame cluster, wherein multiple video frames are divided into multiple keyframe images. The frequency frame is randomly divided into Q video frame sets. Each of the Q video frame sets is clustered into N second video frame clusters. Each second video frame cluster is determined as the target video frame cluster. The similarity between the target video frame cluster and the second video frame clusters in the Q video frame sets is calculated. The second video frame clusters with the highest similarity to the target video frame clusters in each video frame set are merged to obtain the first video frame clusters corresponding to the target video frame clusters. The first video frame clusters corresponding to each second video frame cluster are obtained, resulting in N first video frame clusters corresponding to the N second video frame clusters. The first video frame clusters are determined as a frame cluster, resulting in N frame clusters corresponding to the N first video frame clusters. Image segmentation is performed on each of the keyframe images in the first keyframe image set to obtain image segmentation results for each of the keyframe images in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels. Based on the target multimodal large model, the image segmentation results of the first target task description text and each keyframe image in the first keyframe image set are processed to obtain the first output text; The first output text is converted into speech to obtain the first output audio.

2. The multi-round real-time multimodal large model interaction method according to claim 1, characterized in that, The multi-round real-time multimodal large model interaction method also includes: Acquire a second target audio input by the user and a second target video corresponding to the second target audio, wherein the input time of the second target audio is later than that of the first target audio; The second target audio is converted into text to obtain the second target task description text; A second set of keyframe images is determined based on the second target video, and the second set of keyframe images includes multiple keyframe images in the second target video; Image segmentation is performed on each of the keyframe images in the second keyframe image set to obtain image segmentation results for each of the keyframe images in the second keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels. Based on the target multimodal large model, the second target task description text, the image segmentation results of each keyframe image in the second keyframe image set, the first target task description text, the image segmentation results of each keyframe image in the first keyframe image set, and the first output text are processed to obtain the second output text; The second output text is converted into speech to obtain the second output audio.

3. The multi-round real-time multimodal large model interaction method according to claim 1, characterized in that, The step of determining the third keyframe image set based on the video segment includes: Obtain motion change parameters of the video segment, wherein the larger the motion change parameters, the faster the motion change of the video segment; The number of target images corresponding to the video segment is determined based on the motion change parameters of the video segment, wherein the larger the motion change parameters, the larger the number of target images. The number of video frames representing the target image are extracted from the video segment and used as the third set of keyframe images.

4. The multi-round real-time multimodal large model interaction method according to claim 1, characterized in that, The target multimodal large model includes multiple different encoding modules, modality fusion modules, and decoding modules. The image segmentation results based on the target multimodal large model, processing the first target task description text and each keyframe image in the first keyframe image set, yield a first output text, including: The first target task description text and the image segmentation results of each keyframe image in the first keyframe image set are encoded using different encoding modules in the target multimodal large model to obtain a first feature vector corresponding to the first target task description text and a second feature vector corresponding to the image segmentation results of each keyframe image in the first keyframe image set. The modality fusion module in the target multimodal large model is used to weight and fuse the first feature vector and the second feature vector to obtain a fused feature vector; The fused feature vector is processed using the decoding module in the target multimodal large model to obtain the first output text.

5. A multi-turn real-time multimodal large-scale interactive device, characterized in that, This multi-round, real-time, multimodal, large-scale interactive device includes: The acquisition module is configured to acquire the first target audio input by the user and the first target video corresponding to the first target audio; The first conversion module is configured to convert the first target audio into text to obtain the first target task description text; The determining module is configured to: determine a first set of keyframe images based on the first target video, the first set of keyframe images including multiple keyframe images in the first target video, wherein the first target video is divided into multiple video segments; determine a third set of keyframe images based on the video segments, obtaining multiple sets of third keyframe images corresponding to the multiple video segments; merge the multiple sets of third keyframe images corresponding to the multiple video segments to obtain the first set of keyframe images; cluster multiple video frames in the video segments to obtain multiple frame clusters; and determine the third set of keyframe images based on the center video frame image corresponding to the cluster center of each frame cluster. Multiple video frames are randomly divided into Q video frame sets. Each of the Q video frame sets is clustered into N second video frame clusters. Each second video frame cluster is determined as the target video frame cluster. The similarity between the target video frame cluster and the second video frame clusters in the Q video frame sets is calculated. The second video frame clusters with the highest similarity to the target video frame clusters in each video frame set are merged to obtain the first video frame clusters corresponding to the target video frame clusters. This process yields N first video frame clusters corresponding to the N second video frame clusters. Each first video frame cluster is then determined as a frame cluster, resulting in N frame clusters corresponding to the N first video frame clusters. The segmentation module is configured to perform image segmentation on each of the keyframe images in the first keyframe image set to obtain image segmentation results for each of the keyframe images in the first keyframe image set. The image segmentation results include multiple image segmentation regions and corresponding region labels. The processing module is configured to process the image segmentation results of the first target task description text and each of the keyframe images in the first keyframe image set based on the target multimodal large model to obtain the first output text; The second conversion module is configured to convert the first output text into speech to obtain the first output audio.

6. A computing device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It includes instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-4.

8. A computer program product comprising instructions, the computer program product including program instructions that, when executed on a computer or processor, cause the computer or processor to perform the method as described in any one of claims 1-4.

Images