Method, system and equipment for deep multi-modal expansion of single-modal instruction set

Abstract

The invention discloses a method, a system and equipment for deep multi-modal expansion of a single-modal instruction set. The method comprises the following steps: acquiring an initial instruction and initial single-modal data from an existing instruction data set; acquiring corresponding additional auxiliary information according to the initial single-mode data; performing systematization processing on the initial instruction, integrating a plurality of single-mode data and the additional auxiliary information, and generating text description with semantics; and inputting the text description so as to generate multi-modal data. According to the method, combined processing and interactive analysis are carried out on the multi-modal data, the problem of how to establish an effective relation among the multi-modal information is solved, and the global understanding and decision-making ability of the system for complex tasks is improved. The intelligent level of a multi-modal system can be remarkably improved in multiple application scenes, and particularly, extremely high processing capacity is shown in tasks needing cross-modal data fusion and complex instruction processing.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to a method, system, and device for deep multimodal expansion of a unimodal instruction set. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, unimodal data processing has gradually shown its limitations. Especially when dealing with complex tasks and diverse requirements, a single type of data is difficult to meet the requirements in practical applications. Traditional unimodal instruction sets usually only focus on a single type of data input, such as text, image, or audio. This single-modal processing method has poor adaptability in actual scenarios.

[0003] Current multimodal technologies mostly rely on a single modality or a simple combination between modalities, lacking deep interaction between various modality data. Moreover, existing unimodal instruction sets have limitations in processing multimodal complex instructions. In traditional methods, data of different modalities are often processed independently, resulting in low information utilization efficiency. The traditional multimodal instruction set processing method is often limited to simple alignment or juxtaposition between modalities and cannot achieve collaborative processing and information complementarity of different modality data.

[0004] In view of this, overcoming the technical defects of the above-mentioned existing technologies is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above defects or improvement requirements of the existing technology, the present invention provides a method, system, and device for deep multimodal expansion of a unimodal instruction set, aiming to solve the technical problem of how to ensure the accuracy of instruction semantics and logical consistency.

[0006] To achieve the above object, according to one aspect of the present invention, there is provided a method for deep multimodal expansion of a unimodal instruction set, characterized in that the method includes:

[0007] Obtain an initial instruction and initial unimodal data from an existing instruction dataset;

[0008] Obtain corresponding additional auxiliary information according to the initial unimodal data;

[0009] Systematically process the initial instruction, integrate multiple unimodal data and the additional auxiliary information, and generate a text description with semantics;

[0010] Input the text description to generate multimodal data.

[0011] As a further improvement and supplement to the above solution, the present invention further includes the following additional technical features.

[0012] Preferably, the initial unimodal data includes one of image, text, audio, or video.

[0013] Preferably, the method for obtaining corresponding additional auxiliary information according to the initial unimodal data includes:

[0014] Perform object detection to identify specific objects in the image / video.

[0015] Preferably, the method for obtaining corresponding additional auxiliary information according to the initial unimodal data includes:

[0016] Relate the image or video to the text to generate labels related to the real-world ontology for the objects appearing in the image or video frame.

[0017] Preferably, the method for obtaining corresponding additional auxiliary information according to the initial unimodal data includes:

[0018] Extract semantic information from the audio and generate corresponding text descriptions.

[0019] Preferably, the method for analyzing the initial instruction and generating a refined instruction in combination with the additional auxiliary information includes:

[0020] Extract the information in the initial instruction, combine it with the additional auxiliary information, expand the object details in the initial instruction, and generate a refined instruction.

[0021] Preferably, the method for combining the additional auxiliary information and the refined instruction to output multimodal data includes:

[0022] Convert the text description into visual content, convert multi-frame video data and audio signals into text descriptions, and convert the input text instruction into audio content.

[0023] Preferably, the method for systematically processing the initial instruction, integrating multiple unimodal data, and generating a text description with high semantics includes:

[0024] Extract the basic elements of the initial instruction, combine the additional modal data provided by the additional auxiliary information, and use the additional modal data to add details or context information to the elements in the instruction to generate a text description with high semantics.

[0025] Preferably, the method further includes:

[0026] After generating the multimodal data, automatically evaluate the generated content, determine whether the result of the generated content exceeds the threshold. If it exceeds, pass; if not, perform iteration.

[0027] According to another aspect of the present invention, there is provided a device for deep multi-modal expansion of a single-modal instruction set, the device comprising:

[0028] One or more processors;

[0029] A storage device for storing one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method for deep multi-modal expansion of a single-modal instruction set as described in the first aspect.

[0030] Generally speaking, compared with the prior art by the above technical solution conceived by the present invention, the following beneficial effects are achieved:

[0031] The present invention provides a method, system and device for deep multi-modal expansion of a single-modal instruction set. By constructing a deep fusion mechanism, the problems of association, interaction and complementarity between different modal data are solved, and the unique information of each modal data can be effectively utilized to improve the overall performance. Through the deep expansion of the instruction set, the problem of insufficient ability of the system to handle complex and multi-constraint instructions is solved. At the same time, by enhancing the instruction set from simple to deeply complex, the ability of the system to handle long and difficult instruction scenarios can be effectively enhanced.

[0032] The present invention conducts joint processing and interactive analysis on multi-modal data, solves the problem of how to establish effective connections between multi-modal information, and improves the system's global understanding and decision-making ability for complex tasks. It can significantly improve the intelligence level of multi-modal systems in multiple application scenarios, especially in tasks that require cross-modal data fusion and complex instruction processing, showing extremely strong processing capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly introduced below. Obviously, the following described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0034] Figure 1 It is a schematic flowchart of a method for deep multi-modal expansion of a single-modal instruction set provided in the first embodiment;

[0035] Figure 2 It is a schematic working diagram of a method for deep multi-modal expansion of a single-modal instruction set provided in the first embodiment;

[0036] Figure 3 It is a photo of an airport boarding bridge provided in the first embodiment as an initial instruction;

[0037] Figure 4It is a schematic diagram of a system for deep multi-modal expansion of a single-modal instruction set provided in the second embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of a device for deep multi-modal expansion of a single-modal instruction set provided in the third embodiment of the present invention. Detailed implementation manners

[0039] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0040] Embodiment 1

[0041] With the introduction of multi-modal technology, the integration of different types of data (such as text, images, audio, video, etc.) can significantly improve the overall processing ability of the system.

[0042] Currently, the complexity and depth of instruction sets have become one of the important ways to improve the intelligent performance of the system. By gradually increasing the complexity of instructions and introducing multiple constraints, the accuracy and efficiency of the system in dealing with complex tasks can be effectively improved. At the same time, the multi-modal expansion of single-modal instruction sets has become an important development direction. By fusing and aligning different modal data, the adaptability of the system in complex scenarios can be enhanced, and the processing effect of different types of inputs can be improved.

[0043] However, in existing multi-modal technologies, the integration between different modalities usually stays at the level of simple alignment or parallel processing, or it is difficult to balance the diversity and accuracy of the generated data, resulting in limited generalization ability of the model and inability to achieve deep information interaction. This leads to the lack of effective cooperation ability of multi-modal instruction sets in complex scenarios. Moreover, traditional large language models are prone to logical conflicts or context fragmentation under multi-modal assistance conditions.

[0044] Embodiment 1 of the present invention aims at the problem of insufficient deep integration existing in the multi-modal instruction sets in the prior art, and provides a method for deep multi-modal expansion of a single-modal instruction set. The method includes the steps as Figure 1 shown:

[0045] S101: Obtain initial instructions and initial single-modal data from the existing instruction dataset.

[0046] Obtain initial instructions and initial modal data from existing instruction datasets (such as open-source instruction datasets). These initial instructions are usually basic and simple, aiming to provide a foundation for subsequent complex instruction generation. For example, the instruction may be a text description such as "a picture of a bear catching fish", or it can be a picture / a video / a piece of audio, that is, the modal data can include data in standard formats such as images, texts, audios, and videos. These initial modal data provide the necessary semantic and content basis for subsequent complication, and help improve the generalization ability of the model in multi-modal tasks.

[0047] S102: Obtain corresponding additional auxiliary information according to the initial unimodal data.

[0048] After obtaining the initial modal data, input these initial modal data into an open-source large model for further processing. To expand information, use the following auxiliary models to obtain additional details and context information:

[0049] The GroundingDINO model can be used to identify specific objects in images / videos, such as vehicles, pedestrians, traffic signs, etc., thus providing important object detection information for scene understanding.

[0050] Or, the CLIP model can be used to connect images / videos with texts, and can generate labels related to real-world ontologies for the objects in video frames. For example, it can identify object categories such as "trees", "cats", "buildings", etc., to help the system further understand the image content.

[0051] Or, the PlaceCNN model can be used to identify the background environment in images or videos, such as urban streets, shopping malls, parks, etc. It provides rich semantic information for understanding the scene background and improves the system's spatial perception ability in complex scenes.

[0052] Or, the RECAP model can be used to extract semantic information from audio and generate corresponding text descriptions. It is applicable to speech recognition and the generation of conversations and sound descriptions in scenes, especially in multi-modal tasks, which can help achieve the linked processing of audio and text.

[0053] Or, the SenseVoice model can be used to identify specific events in audio, such as "vehicle driving", "horn sound", "crowd conversation", etc., and generate labels for audio segments to facilitate the further linked processing of audio with video and text.

[0054] S103: Systematically process the initial instructions, integrate multiple unimodal data and the additional auxiliary information, and generate a text description with semantics.

[0055] Systematically complicate the initial instructions through large language models, effectively integrate multimodal information, and generate text descriptions with high semantics.

[0056] In the process of multimodal data processing, large language models (LLMs) play a key role in deepening and complicating instructions. They can not only handle basic instructions but also generate more complex instruction texts by introducing multi-step reasoning, adding constraints, and combining multimodal auxiliary information. Their core advantage lies in the layer-by-layer decomposition of complex tasks, the integration of multimodal information, and the generation of text descriptions with high semantic richness.

[0057] When processing simple instructions, LLMs first analyze the basic semantic structure of the instructions and then perform multi-level complication operations according to the preset task scenarios. For example, for the initial instruction "There are vehicles moving on the street" generated by a simple model, LLMs first extract the basic elements such as "vehicles" and "street", and combine the visual and audio data provided by the auxiliary model to further explore deeper semantic information. LLMs can utilize object recognition information from GroundingDINO and CLIP, etc., to further enrich the object details in the original instructions. For example, for "There are vehicles moving on the street", LLMs can introduce more specific object descriptions based on visual detection results: "A red sedan is moving along the left lane, while several pedestrians are waiting for the red light at the intersection." This complication not only adds details such as the color and type of the vehicle but also combines the position and actions of the pedestrians, further enriching the scene description. Eventually, it expands to, for example: "Near the crossroads, a red sedan is driving past from the left lane. Not far in front of the car, several pedestrians are waiting for the red light and preparing to cross the road. The traffic flow orderly makes way for them to ensure the safety of the pedestrians. In the distance, the background is three tall commercial office buildings, forming a typical urban landscape." This complicated description provides more detailed support for subsequent multimodal generation.

[0058] The core idea of expanding the modality is to input the complicated instructions and multimodal auxiliary information into an open-source generation model to obtain the corresponding multimodal output. For example, based on multimodal inputs of vision, text, and audio, corresponding images, text descriptions, audio commentaries, or video clips can be generated. Using the instructions deepened by LLMs, the generated multimodal data not only has high accuracy and consistency but also can meet the information conversion and generation requirements in complex scenarios.

[0059] (1) Use StableDiffusion / GPT4o to convert text descriptions into visual content. For example, when the input instruction is "Generate an image of a sunset over a lake", it can not only generate static scenes but also handle the changes in complex lighting, textures, and colors, making the generated images highly realistic and visually impactful.

[0060] (2) Use the Allegro video generation model, combined with extended visual and audio encoders, to convert multi-frame video data and audio signals into text descriptions, supporting complex task reasoning. When the input is something like "Generate a video showing a busy urban traffic scene", Allegro can generate a video including elements such as pedestrians, vehicles, and traffic lights, and provide a complete multi-modal description in combination with audio information.

[0061] Use Stable-Audio-Open to generate streaming audio content according to the input text instruction. For example, when receiving the instruction "Generate a voice commentary on the traffic situation", Stable-Audio-Open can quickly call its audio generation module to synthesize and output clear and natural audio in real time, describing the urban traffic situation, such as "The current street has heavy traffic. It is recommended to choose an alternative route to avoid traffic congestion". This technology demonstrates excellent performance in scenarios that require immediate response, providing users with efficient and accurate audio commentary.

[0062] S104: Input the text description to generate multi-modal data.

[0063] After generating the multi-modal data, automatically evaluate the generated content to determine whether the result of the generated content exceeds the threshold. If it exceeds, it passes; if not, iterate.

[0064] Use reinforcement learning algorithms, combined with human feedback (RLHF) technology, to build an automated model for evaluating the generated content. This model deeply integrates multi-modal information with complex instructions to optimize the generated tasks and instruction sets round by round.

[0065] In each round of the iterative process, the system not only deeply processes the generated instructions and multi-modal tasks but also validates them through the automatic evaluation model to ensure that the generated content meets the expected requirements in terms of logic, rationality, complexity, etc. By introducing multi-dimensional verification of multi-modal instructions, the system can effectively improve the executability of the instructions and the quality of the generated content. In addition, the introduction of RLHF technology enables the system to be trained and optimized according to the actual user feedback data, thus significantly improving the intelligent level of the system.

[0066] The optimization mechanism of multi-round iteration and screening allows the system to continuously adjust and improve the processing strategy in each round of task generation. For tasks that fail to generate or do not meet expectations, the system can automatically detect problems and perform targeted optimization. This optimization process is driven by the reward signal in reinforcement learning, enabling the system to gradually enhance its coping ability in the face of complex scenarios, minimize the probability of task failure, and improve execution efficiency. At the same time, this mechanism ensures that the system can flexibly respond to various task requirements, thus demonstrating extremely high adaptability and intelligence in the complex environment of multi-modal fusion.

[0067] By introducing the reinforcement learning algorithm, the present invention can continuously optimize the multi-modal generation task in the multi-round generation and screening process, ensuring that the generated instructions and content are more reasonable and complex, and improving the overall execution efficiency and task quality of the system. Specifically, through the action value function (i.e., the trained RLHF model), we can determine whether a newly generated answer passes. Suppose we are iteratively generating an answer to a text-modal question, where the state represents the text currently received by the system, such as "There are a switch and a router in the picture."; the action represents the answer strategy selected by the system, such as generating "It seems to be a network topology diagram. There are multiple device nodes in the diagram, including a switch (Switch) and a router (Router). The devices are interconnected through different connections (such as virtual local area network VLAN or physical lines), and may involve different subnet configurations or routing paths." Then, the calculated function value represents the expected reward under the given question and answer conditions. If the obtained value exceeds the preset threshold, then this answer is considered qualified; if it is lower than this threshold, the answer is considered to have failed and needs further iteration.

[0068] The method for obtaining corresponding additional auxiliary information according to the initial single-modal data includes:

[0069] As Figure 2 described, object detection is performed to identify specific objects in the image / video. For example, the GroundingDINO model is used to identify specific objects in the image / video, such as vehicles, pedestrians, traffic signs, etc., thereby providing important object detection information for scene understanding.

[0070] In the first embodiment, the method for obtaining corresponding additional auxiliary information according to the initial single-modal data includes:

[0071] As Figure 2As described above, an image or video is associated with text to generate tags related to real-world entities for the objects appearing in the image or video frame. For example, the CLIP model can be used to associate an image / video with text, and tags related to real-world entities can be generated for the objects in the video frame. For example, it can identify object categories such as "trees", "cats", "buildings", etc., helping the system to further understand the image content.

[0072] In the first embodiment, the method for obtaining corresponding additional auxiliary information according to the initial unimodal data includes:

[0073] As Figure 2 described above, semantic information is extracted from the audio and a corresponding text description is generated. For example, the PlaceCNN model is used to identify the background environment in an image or video, such as a city street, a shopping mall, a park, etc. It provides rich semantic information for understanding the scene background and enhances the system's spatial perception ability in complex scenes.

[0074] In the first embodiment, the method for analyzing the initial instruction and generating a refined instruction in combination with the additional auxiliary information includes:

[0075] As Figure 2 described above, the information in the initial instruction is extracted, combined with the additional auxiliary information, and the object details in the initial instruction are extended to generate a refined instruction. For example, the RECAP model is used to extract semantic information from the audio and generate a corresponding text description. It is applicable to speech recognition and the generation of conversations and sound descriptions in a scene, especially helping to achieve the linked processing of audio and text in multimodal tasks.

[0076] In the first embodiment, the method for combining the additional auxiliary information and the refined instruction to output multimodal data includes:

[0077] As Figure 2 described above, the text description is converted into visual content, the multi-frame video data and audio signal are converted into text descriptions, and the input text instruction is converted into audio content. For example, the SenseVoice model is used to identify specific events in the audio, such as "vehicle driving", "horn sound", "crowd conversation", etc., and generate tags for the audio segments, facilitating the further linked processing of audio with video and text.

[0078] In the first embodiment, the method for systematically processing the initial instruction, integrating multiple unimodal data, and generating a text description with high semantics includes:

[0079] The basic elements of the initial instruction are extracted, combined with the additional modal data provided by the additional auxiliary information, and the additional modal data is used to add details or context information to the elements in the instruction, generating a text description with high semantics.

[0080] In this embodiment 1, the method further includes:

[0081] After generating multimodal data, the generated content is automatically evaluated to determine whether the result of the generated content exceeds the threshold. If so, it passes; if not, it iterates.

[0082] In the first embodiment of the present invention, the initial modal data is first obtained through an open source instruction dataset, and multiple auxiliary models are used to process images, videos, and audio to extract detailed object information and scene background, which provides the necessary foundation for deep data analysis. Furthermore, the initial instructions are systematically and complicatedly processed through a large language model, effectively integrating multimodal information to generate highly semantic text descriptions. In order to ensure the accuracy and consistency of the generated results, multiple open source generation models are also integrated to convert complex text descriptions into corresponding image, video, and audio outputs. Finally, multiple rounds of iterations and reinforcement learning (RLHF) technology are combined to continuously adjust and optimize the generation process through sophisticated automated evaluation and user feedback to ensure that the system can show excellent adaptability and efficiency when dealing with complex and changing application scenarios. This process reflects the deep mining and continuous efforts in technology.

[0083] If the initial command input is a photo of an airport bridge, Figure 3 Said:

[0084] The image data is sent to the information extraction auxiliary model. At this time, because it is an image, only GroundingDINO, CLIP and PlaceCNN will extract information, namely the specific objects in the image, the background environment and other information.

[0085] The initial image and the extracted auxiliary information are applied to the input templates of different modes defined in advance, and passed into LLM to obtain the first text information. Then, new alignment instructions are generated for each mode, such as StableDiffusion generates "a corridor with lights on, and a logistics worker standing next to it, carrying maintenance tools"; Allegro generates "a corridor with lights on, and a maintenance worker walks by with a maintenance toolbox"; StableAudio generates "a corridor with lights on, and a maintenance worker walks by with a maintenance toolbox, and the echo of footsteps spreads in the corridor", so that LLM itself can generate a deep and complex text description of the image.

[0086] The generated new modal data is formatted and passed to the pre-trained RLHF model to judge the data quality. For example, the image modality is the action value function Q(s,a) which represents the expected reward for generating behavior a (generating a picture of "a lighted corridor with a maintenance toolbox in the middle and a logistics staff standing next to it") under a given state s (here: "a corridor with a light on, a maintenance toolbox in the middle and a logistics staff standing next to it").

[0087] The audio modality is given the state of "a lighted corridor bridge, a maintenance worker walks by with a maintenance toolbox, and the echo of his footsteps spreads in the corridor bridge", and the reward value of generating behavior a is calculated.

[0088] For the video modality and the audio modality, the given state is: "A lighted covered bridge, a maintenance worker walks by with a maintenance toolbox", and the reward value of generating behavior a is calculated.

[0089] The RLHF model can also be used in the community open source, which has been trained through a large amount of data. If the score does not meet the standard, it will continue to be iterated, and if it meets the standard, the data will be successfully expanded.

[0090] Through the deep expansion of the multimodal instruction set, this embodiment realizes the deep integration and interaction of different modal data (such as text, images, audio, video, etc.), so that the system can fully utilize the advantages of various types of data and improve the ability to handle complex tasks. This integration can not only realize the collaborative processing of multimodal information, but also improve the adaptability and decision-making ability of the system when dealing with complex tasks.

[0091] This embodiment introduces a multi-level complex instruction processing mechanism, which can effectively process complex instruction sets with multiple constraints, significantly improving the system's processing accuracy and efficiency for long instructions and multi-step tasks. By making the instruction set more complex and in-depth, the system has stronger intelligent response capabilities, especially for multimodal tasks in complex scenarios.

[0092] The technical solution of this embodiment is not only applicable to traditional single-modal command processing, but also can demonstrate remarkable intelligent performance in multiple application scenarios such as cross-modal data processing, complex scene analysis, multimodal reasoning, etc., especially in fields that require multimodal fusion and complex command execution. It has broad application prospects.

[0093] Through these technical advantages, this embodiment can significantly improve the intelligence level of large models in multimodal tasks, enabling them to have higher processing efficiency and flexibility when dealing with complex tasks.

[0094] Embodiment 2:

[0095] This embodiment 2 provides a system for deeply extending a single-mode instruction set to multi-modality, such as Figure 4 As shown, the system includes:

[0096] An initial module for obtaining initial instructions and initial unimodal data from an existing instruction dataset;

[0097] An auxiliary module for obtaining corresponding additional auxiliary information based on the initial unimodal data;

[0098] An expansion module for systematically processing the initial instructions, integrating multiple unimodal data and the additional auxiliary information to generate a semantic text description;

[0099] A generation module for inputting the text description to generate multimodal data;

[0100] An evaluation module for automatically evaluating the generated content of the multimodal data, determining whether the result of the generated content exceeds a threshold. If it exceeds, it passes; if it does not exceed, iteration is performed.

[0101] By constructing a deep fusion mechanism, the problems of association, interaction, and complementarity between different modal data are solved, and the unique information of each modal data can be effectively utilized to improve the overall performance. Through the deep expansion of the instruction set, the problem of insufficient ability of the system to handle complex instructions with multiple constraints is solved. At the same time, by enhancing the instruction set from simple to deeply complex, the system's ability to handle long and difficult instruction scenarios can be effectively enhanced. The present invention performs joint processing and interactive analysis on multimodal data, solves the problem of how to establish effective connections between multimodal information, and improves the system's global understanding and decision-making ability for complex tasks.

[0102] This embodiment can significantly improve the intelligent level of the multimodal system in multiple application scenarios, especially in tasks that require cross-modal data fusion and complex instruction processing, showing extremely strong processing capabilities.

[0103] Embodiment Three:

[0104] A device for deep multimodal expansion of a unimodal instruction set, as Figure 5 shown, the device includes:

[0105] One or more processors;

[0106] A storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method for deep multimodal expansion of a unimodal instruction set as described in any one of Embodiment One.

[0107] Figure 5 This is the structural schematic diagram of the device for deep multimodal expansion of a unimodal instruction set provided in Embodiment Three. Figure 5The figure shows a block diagram of a device suitable for implementing the exemplary single-modal instruction set deep multi-modal expansion according to the embodiments of the present invention. Figure 5 The device shown for the single-modal instruction set deep multi-modal expansion is merely an example and should not impose any limitation on the functions and scope of use of the embodiments of the present invention.

[0108] As Figure 5 shown, the device for the single-modal instruction set deep multi-modal expansion is presented in the form of a general-purpose device. The components of the device for the single-modal instruction set deep multi-modal expansion may include, but are not limited to: one or more processors or processing units, a memory, and a bus connecting different system components (including the memory and the processing unit).

[0109] The bus represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0110] The device for the single-modal instruction set deep multi-modal expansion typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by a device capable of being modified by an intelligent logging interpretation model, including volatile and non-volatile media, removable and non-removable media.

[0111] The memory may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory. The device for the single-modal instruction set deep multi-modal expansion may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system may be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 5 not shown, commonly referred to as a "hard disk drive"). Although Figure 5 not shown in the figure, a disk drive for reading and writing on a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing on a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus through one or more data media interfaces. The memory may include at least one program product having a set (such as at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0112] A program / util utility having a set (at least one) of program modules can be stored, for example, in a memory. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules generally execute the functions and / or methods in the embodiments described in the present invention.

[0113] The device for expanding the depth of a single-modal instruction set into a multi-modal one can also communicate with one or more external devices (such as a keyboard, a pointing device, a display, etc.), and can also communicate with one or more devices that enable a user to interact with the device for expanding the depth of the single-modal instruction set into a multi-modal one, and / or communicate with any device (such as a network card, a modem, etc.) that enables the device for expanding the depth of the single-modal instruction set into a multi-modal one to communicate with one or more other devices. Such communication can be carried out through an input / output (I / O) interface. Moreover, the device for correcting the intelligent logging interpretation model can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter. As Figure 5 shown, the network adapter communicates with other modules of the device for expanding the depth of the single-modal instruction set into a multi-modal one through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the device for expanding the depth of the single-modal instruction set into a multi-modal one, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0114] The processing unit executes various functional applications and data processing by running programs stored in the memory, for example, implementing the method for expanding the depth of the single-modal instruction set into a multi-modal one provided by any embodiment of the present invention. That is: obtaining initial instructions and initial single-modal data from an existing instruction dataset; obtaining corresponding additional auxiliary information according to the initial single-modal data; performing systematic processing on the initial instructions, integrating multiple single-modal data and the additional auxiliary information, generating a text description with semantics; inputting the text description to generate multi-modal data.

[0115] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for deeply extending a single-mode instruction set to multi-modality, characterized in that: The method comprises: Obtaining initial instructions and initial single-modality data from an existing instruction data set; Acquiring corresponding additional auxiliary information according to the initial single-modal data; Systematically processing the initial instruction, integrating multiple single-modal data and the additional auxiliary information, and generating a text description with semantics; The text description is input to generate multimodal data.

2. The method for deep multimodal expansion of a single-mode instruction set according to claim 1, characterized in that: The initial unimodal data includes: one of images, texts, audio or video.

3. The method for deep multimodal expansion of a single-modal instruction set according to claim 1, characterized in that: The method for acquiring corresponding additional auxiliary information according to the initial single-modal data includes: Perform object detection to identify specific objects in images / videos.

4. The method for deep multimodal expansion of a single-modal instruction set according to claim 3, characterized in that: The method for acquiring corresponding additional auxiliary information according to the initial single-modal data further includes: Link images or videos to text and generate labels related to real-world ontology for objects appearing in images or video frames.

5. The method for deep multimodal expansion of a single-modal instruction set according to claim 4, characterized in that: The method for acquiring corresponding additional auxiliary information according to the initial single-modal data further includes: Extract semantic information from audio and generate corresponding text description.

6. The method for deep multimodal expansion of a single-modal instruction set according to claim 1, characterized in that: The method of analyzing the initial instruction and generating a deepening instruction in combination with the additional auxiliary information includes: Extract the basic elements of the initial instruction, combine them with the additional modal data provided by the additional auxiliary information, use the additional modal data to add details or context information to the elements in the instruction, and generate a text description with semantics.

7. The method for deep multimodal expansion of a single-modal instruction set according to claim 6, characterized in that: The method of combining the additional auxiliary information with the deepening instruction to output multimodal data includes: Convert text descriptions into visual content, convert multi-frame video data and audio signals into text descriptions, and convert input text instructions into audio content.

8. The method for deep multimodal expansion of a single-modal instruction set according to claim 1, characterized in that: The method further comprises: After generating multimodal data, the generated content is automatically evaluated to determine whether the result of the generated content exceeds the threshold. If so, it passes; if not, it iterates.

9. A system for deep multimodal expansion of a single-mode instruction set, characterized in that the system include: An initial module, used to obtain initial instructions and initial single-modal data from an existing instruction data set; Auxiliary module: used for acquiring corresponding additional auxiliary information according to the initial single-modal data; Extension module: used for systematically processing the initial instruction, integrating multiple single-modal data and the additional auxiliary information, and generating a text description with semantics; A generation module: used for inputting the text description to generate multimodal data; Evaluation module: used to automatically evaluate the generated content of the multimodal data, and determine whether the result of the generated content exceeds a threshold. If so, it is passed; if not, it is iterated.

10. A device for deep multimodal expansion of a single-mode instruction set, characterized in that the device include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for deep multimodal expansion of a single-modal instruction set as described in any one of claims 1-8.

Images