Method and device for constructing multi-modal perception and cognition data set

Abstract

The invention provides a multi-mode perception and cognition data set construction method and device, and the method comprises the steps: obtaining image information which comprises a picture captured by a camera of an intelligent agent in real time; constructing a knowledge graph according to the image information; wherein the knowledge graph comprises entity information and a position relationship between entities; segmenting the image information into sub-region images which are not overlapped; according to the method, image description is performed according to the sub-region images and the knowledge graph, a natural language text describing image content is generated, and a multi-modal perception and cognition data set is constructed according to the natural language text, so that the problems of high cost, low efficiency and low accuracy of a multi-modal data set labeling method in the prior art are solved; and the perception ability of the multi-modal model and the advanced cognitive ability including reasoning, comparison and description are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for constructing a multimodal perception and cognitive data set. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal datasets are playing an increasingly important role in computer vision, natural language processing, speech recognition and other fields. Multimodal datasets refer to data sets containing multiple modalities (such as text, images, audio, etc.). By effectively integrating and analyzing these data, the performance of artificial intelligence models can be greatly improved.

[0003] Most of the existing multimodal datasets rely on manual crowdsourcing annotation. The process of manual crowdsourcing annotation mainly includes the following steps: the data provider publishes the original data to the crowdsourcing platform; the crowdsourcing participants annotate the data; the data provider reviews and corrects the annotation results.

[0004] However, this method that relies on manual crowdsourcing annotation has the following disadvantages: Huge cost: Manual crowdsourcing annotation requires paying participants a certain amount of compensation. As the amount of data increases, the annotation cost continues to rise; Time-consuming communication: During the annotation process, the data provider needs to communicate with the participants several times to ensure the quality of the annotation, which undoubtedly increases the project cycle; Information comprehension error: Due to differences in participants’ background knowledge, cognitive abilities, etc., there are certain errors in the annotation results, which in turn affects the quality of the data set.

[0005] In summary, the existing multimodal dataset annotation methods have certain limitations, and an efficient, low-cost, and highly accurate automatic annotation method is urgently needed to solve this problem. Summary of the invention

[0006] In view of this, the technical problem to be solved by the present invention is to provide a method, device, electronic device, computer-readable storage medium and computer program product for constructing a multimodal perception and cognitive dataset, so as to solve the problems of high cost, low efficiency and low accuracy of multimodal dataset annotation methods in the prior art.

[0007] In one aspect, the present invention provides a method for constructing a multimodal perception and cognition dataset, characterized in that the method comprises: Acquire image information; wherein the image information includes a picture captured in real time by the camera of the intelligent agent itself; Constructing a knowledge graph based on the image information; wherein the knowledge graph includes entity information and positional relationships between entities; Dividing the image information into non-overlapping sub-region images; An image description is performed based on the sub-region image and the knowledge graph, a natural language text describing the image content is generated, and a multimodal perception and cognition dataset is constructed based on the natural language text.

[0008] Optionally, dividing the image information into non-overlapping sub-region images comprises the following steps: S1: converting the image information into a pixel feature matrix, wherein each pixel point is regarded as a sample, and normalizing the pixel features to obtain a processed image; S2: Determine the number of clusters according to the elbow rule and the silhouette coefficient; where the number of clusters represents the number of sub-region images; S3: Randomly select the initial cluster center through the K-means clustering algorithm; S4: For each pixel in the image, calculate its Euclidean distance to each initial cluster center, and assign it to the cluster represented by the nearest initial cluster center; S5: Calculate the mean of all points in each cluster by using the K-means clustering algorithm, and recalculate the cluster center of each cluster according to the mean; Steps S3, S4, and S5 are repeatedly performed until the change value of the cluster center of each cluster is less than a predetermined threshold, or the number of iterations reaches a preset number of iterations, or the change of samples within the cluster is less than a preset change amount.

[0009] Optionally, before constructing the knowledge graph according to the image information, the method further includes: Obtaining coarse-grained information of the image information; wherein granularity is used to describe the granularity level of image processing or analysis, and the coarse-grained information is obtained by performing global processing on the image, focusing on the characteristics of the overall image; The labels of the coarse-grained information are refined through a visual target detection model to obtain fine-grained entity information and the positional relationship between fine-grained entities; wherein fine-grained means local subclass division and recognition of the image; the positional relationship between fine-grained entities includes up, down, left, right, front, back, inside, and outside.

[0010] Optionally, constructing a knowledge graph according to the image information includes: converting the image information into language information; Using the convolutional architecture caffee for fast feature embedding of computer vision models as an information detection and extraction model, the language information is overlapped with the preset known information to obtain a calculation result; Supplementing the entity information according to the calculation result to obtain supplemented language information; The supplemented language information is converted into subject-verb-object (SPO) triple information to construct the knowledge graph.

[0011] Optionally, after constructing the knowledge graph according to the image information, the method further includes: Collect the subject-verb-object (SPO) triple sample data of the input entity; The Qwen-1.5-7b pre-trained model is used as the basic model; The basic model is fine-tuned using the parameter efficient fine-tuning technology LoRA technology; wherein the training parameters include a learning rate (lr) of 1e-3 and a training round (epoch) of 10; Using the subject-verb-object SPO triple sample data to train the basic model to obtain a trained model; The trained model is used to transform subject-verb-object (SPO) triple information into information with fluent and rich natural language description.

[0012] Optionally, performing image description according to the sub-region image and the knowledge graph, generating natural language text describing the image content, and constructing a multimodal perception and cognition dataset according to the natural language text includes: Call the multimodal artificial intelligence model GPT-4o to generate descriptive text for the input sub-region image; The descriptive text is combined with the information description of the central entity and input into the language model LLM to generate single-hop question-answer pairs and multi-hop question-answer pairs; wherein the single-hop question-answer pairs refer to questions and answers directly based on the image description, while the multi-hop question-answer pairs refer to questions and answers that require reasoning or additional information processing.

[0013] In a second aspect, the present invention further provides a device for constructing a multimodal perception and cognitive data set, the device comprising: An acquisition module, used to acquire image information; wherein the image information includes a picture captured in real time by the camera of the intelligent agent itself; A first construction module is used to construct a knowledge graph based on the image information; wherein the knowledge graph includes entity information and positional relationships between entities; A segmentation module, used for segmenting the image information into non-overlapping sub-region images; The second construction module is used to describe the image based on the sub-region image and the knowledge graph, generate natural language text describing the image content, and construct a multimodal perception and cognition dataset based on the natural language text.

[0014] Optionally, the first building block includes: A conversion unit, used for converting the image information into language information; A computing unit, used for using a convolutional architecture caffee for fast feature embedding of a computer vision model as an information detection and extraction model, calculating the overlap between the language information and preset known information, and obtaining a calculation result; A supplementing unit, used to supplement the entity information according to the calculation result to obtain supplemented language information; A construction unit is used to convert the supplemented language information into subject-verb-object (SPO) triple information to construct the knowledge graph.

[0015] Optionally, the device further comprises: A collection module is used to collect subject-verb-object SPO triple sample data of input entities; Select the module to use the Qwen-1.5-7b pre-trained model as the base model; A fine-tuning module, used to fine-tune the basic model using the parameter efficient fine-tuning technology LoRA technology; wherein the training parameters include a learning rate (lr) of 1e-3 and a training round (epoch) of 10; A training module, used to train the basic model using the subject-verb-object SPO triple sample data to obtain a trained model; The conversion module is used to convert the subject-verb-object (SPO) triple information into information with fluent and rich natural language description through the trained model.

[0016] Optionally, the second building block includes: A first generating unit is used to call the multimodal artificial intelligence model GPT-4o to generate descriptive text for the input sub-region image; The second generation unit is used to combine the descriptive text with the information description of the central entity and input it into the language model LLM to generate single-hop question-answer pairs and multi-hop question-answer pairs; wherein the single-hop question-answer pairs refer to questions and answers directly based on the image description, and the multi-hop question-answer pairs refer to questions and answers that require reasoning or additional information processing.

[0017] According to a third aspect of the present invention, there is provided an electronic device, comprising: Processor; and Memory for storing programs, Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes the method for constructing a multimodal perception and cognition dataset according to the first aspect of the present invention.

[0018] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method for constructing a multimodal perception and cognition dataset according to the first aspect of the present invention.

[0019] In a fifth aspect, the present invention further provides a computer program product, the computer program product comprising instructions, which, when executed, causes a computer to execute the method for constructing a multimodal perception and cognitive data set according to the first aspect of the present invention. Through the embodiments of the present invention, image information is obtained, and the image information includes a picture captured by the camera of the intelligent body itself in real time; a knowledge graph is constructed according to the image information; wherein the knowledge graph includes entity information and positional relationships between entities; the image information is divided into non-overlapping sub-region images; the image is described according to the sub-region images and the knowledge graph, a natural language text describing the image content is generated, and a multimodal perception and cognitive data set is constructed according to the natural language text, which solves the problems of high cost, low efficiency and low accuracy of multimodal data set annotation methods in the prior art, and improves the perception ability of the multimodal model as well as the advanced cognitive ability including reasoning, comparison and description.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings below are part of the specification of the present invention, and illustrate exemplary embodiments of the present invention. Together with the description of the specification, the accompanying drawings are used to explain the principles of the present invention.

[0022] Figure 1 It is a flowchart of a method for constructing a multimodal perception and cognition dataset according to an embodiment of the present invention.

[0023] Figure 2 It is a structural block diagram of a device for constructing a multimodal perception and cognitive data set according to an embodiment of the present invention.

[0024] Figure 3 is a structural block diagram of a first building module according to an embodiment of the present invention.

[0025] Figure 4 4 is another structural block diagram of a device for constructing a multimodal perception and cognitive dataset according to an embodiment of the present invention.

[0026] Figure 5 is a structural block diagram of a second building module according to an embodiment of the present invention.

[0027] Figure 6A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0029] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0030] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0031] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0032] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0033] As used herein, "and / or" includes any or all combinations of the items described.

[0034] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0035] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0036] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0037] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0038] The embodiments of the present invention define perception and cognition as follows: In the field of computer vision (CV), perception and cognition are two closely related but distinct concepts: 1) Perception: In computer vision, perception usually refers to receiving and processing information from the outside world, such as images or videos, through the visual system. This includes basic image processing, such as edge detection, feature extraction, etc., as well as preliminary recognition of objects. Perception is the basis of cognition, which involves the process of organizing, recognizing and interpreting visual information. 2) Cognition: Cognition is more complex, and it includes not only perception, but also further processing and understanding of perceived information. In the field of computer vision, cognition involves the classification, recognition, scene understanding and deeper semantic interpretation of objects. The cognitive process includes aspects such as knowledge formation, attention, memory, reasoning, problem solving, decision making and language generation.

[0039] The difference between perception and cognition is mainly reflected in the following aspects: Depth of information processing: Perception focuses more on the initial processing of raw visual information, while cognition involves in-depth analysis and understanding of this information.

[0040] Complexity of information processing: Perception is the input of information, and cognition is the processing and output of information. Perception is a more direct and basic stage, while cognition involves a more complex information processing process.

[0041] Subjective and objective: Perception is objective and is a direct reflection of external information; cognition is subjective and involves the individual's further processing of sensory signals, including concept formation, judgment, and problem solving.

[0042] Concrete vs. Abstract: Perception is concrete and is directly related to the information received by the senses; cognition is abstract and involves further abstraction and conceptualization of the perceived information.

[0043] In general, in the field of computer vision, perception and cognition together constitute the complete processing flow of visual information, from the initial image acquisition and processing to the final understanding and interpretation.

[0044] In this embodiment, a method for constructing a multimodal perception and cognition dataset is provided. Figure 1 is a flowchart of a method for constructing a multimodal perception and cognitive dataset according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps. It should be noted that the supplementary numbers do not represent the execution order of the steps. The execution order between the steps can be changed without affecting the implementation of the overall solution.

[0045] Step S101: Obtain image information; the image information includes the real-time captured images by the agent's own camera. For example, since the images of the data in this data set can be directly captured by the UE interface, it naturally has simple information (coordinates, rough labels) of these assets, and this data is used as the initial data state.

[0046] Step S102: Construct a knowledge graph based on the image information; wherein the knowledge graph includes entity information and positional relationships between entities. Specifically, similar to a game interface, the content of the picture is the picture and coarse-grained information captured in real time by the camera of the intelligent body itself. For example, the position coordinates of entities such as a table, a mirror, a bed and some toys in the current room, and the information of color and shape can be obtained in real time. It should be noted that in the field of artificial intelligence, the definitions of coarse labels and fine labels of images are as follows: Coarse labels, also called rough labels or coarse-grained labels, are a type of label used in deep learning. It does not require accurate classification of each sample, but by macro-grasping the sample set, assigning labels with a certain degree of coarseness to the sample. In multi-category classification problems, coarse labels are labels of the category assigned to part of the sample set. The accuracy of these labels is not required to be very high, and it only needs to roughly reflect the category to which the sample belongs. Fine labels, or fine-grained labels, refer to the process of labeling a dataset. According to the characteristics and usage requirements of the dataset, a relatively fine set of labeling rules is artificially designed, and then each sample in the dataset is assigned a label according to these rules. Compared with coarse labels, fine labels are more detailed and accurate, and the requirements for labelers are relatively high. Therefore, the production cost of fine labels is higher, and they are suitable for datasets that require high-precision classification. In image recognition, fine labels may involve the classification of specific objects in the image, such as different types of birds or car models, which require more detailed distinctions. In summary, coarse labels provide a macroscopic classification of image content, while fine labels provide more detailed and specific classification information. Both play an important role in deep learning and computer vision tasks. The choice of which type of label to use depends on the specific application requirements and resource conditions. For example, the definition of coarse labels, since the names of these data assets are not completely correct, such as a ball is actually a basketball, but the initial data assets only have the name of the general category labeled as: ball. Therefore, it is a coarse-grained information; the definition of fine label ground truth refers to the real and fine-grained name of an entity that cannot be further refined, such as: basketball.

[0047] Step S103: Divide the image information into non-overlapping sub-region images. Since a picture may contain a large number of entities, if all of this information is sent to the LLM at the same time, it will result in a large number of input tokens and fail to generate high-quality data. Therefore, the picture is divided into non-overlapping Region Graphs. This ensures high quality and accuracy of the generated data based on the information clipping.

[0048] Step S104: Describe the image based on the sub-region image and the knowledge graph, generate natural language text describing the image content, and construct a multimodal perception and cognitive dataset based on the natural language text. In an optional embodiment, image information extraction, at this stage, can guide GPT-4o to summarize the image and use this summary as the language description of the image.

[0049] Through the above steps, the reliance on manual labeling is significantly reduced, thereby reducing the overall labeling cost; the speed and processing efficiency of data labeling are improved through automated and intelligent labeling technology; the accuracy of labeling can be improved, providing higher quality training data for multimodal models; the multimodal model's ability to perceive data is improved, enabling it to more accurately understand and process information in different modalities such as images and texts; not only the perception ability is improved, but also the model's high-level cognitive abilities such as reasoning, comparison and description are enhanced, making the model perform better when handling complex tasks.

[0050] Regarding the specific implementation of the above step S103, in an optional embodiment, dividing the image information into non-overlapping sub-region images includes the following steps: S1: The image information is converted into a pixel feature matrix, where each pixel is taken as a sample, and the pixel features are normalized to obtain the processed image. This ensures that the features of each pixel are accurately captured and processed, thereby improving the accuracy of subsequent sub-region image segmentation.

[0051] S2: Determine the number of clusters according to the elbow rule and the silhouette coefficient; the number of clusters represents the number of sub-region images. The optimal number of sub-region images can be automatically and effectively determined, avoiding the subjectivity and inaccuracy of manual experience judgment.

[0052] S3: The initial cluster center is randomly selected through the K-means clustering algorithm, which provides a stable starting point for the clustering process and helps improve the stability and reliability of the clustering results.

[0053] S4: For each pixel in the image, the Euclidean distance between it and each initial cluster center is calculated, and it is assigned to the cluster represented by the nearest initial cluster center, ensuring the accuracy and rationality of pixel assignment.

[0054] S5: The K-means clustering algorithm is used to calculate the mean of all points in each cluster, and the cluster center of each cluster is recalculated based on the mean. This ensures the stability of the clustering process, reduces fluctuations during the algorithm operation, and improves the convergence speed and stability of the algorithm.

[0055] Steps S3, S4, and S5 are repeated until the change value of the cluster center of each cluster is less than a predetermined threshold, or the number of iterations reaches a preset number of iterations, or the change of samples within the cluster is less than a preset change amount. This achieves iterative optimization of the clustering process and ensures the optimality of the final sub-region image segmentation result.

[0056] Through the precise sub-region image segmentation method of this optional embodiment, the generalization ability of the multimodal model for different image features is enhanced, making it more effective in processing complex image information.

[0057] In the process of extracting structured information, in an optional embodiment, the coarse-grained information of the image information is obtained, wherein the granularity is used to describe the granularity level of image processing or analysis, and the coarse-grained information is obtained by global processing on the image, focusing on the characteristics of the overall image. The labels of the coarse-grained information are refined by the visual target detection model to obtain fine-grained entity information and the positional relationship between the fine-grained entities; wherein the fine-grained represents the local subclass division and recognition of the image; the positional relationship between the fine-grained entities includes up, down, left, right, front, back, inside, and outside. Specifically, according to the obtained coarse-grained information, relevant model calculations are performed to finally obtain fine-grained structured information, where the model includes using the mainstream visual target detection model to refine the coarse-grained labels, such as refining the ball into a basketball, and using the position calculation tool to calculate the positional relationship of each entity, and the positional relationship is specific: up, down, left, right, front, back, inside, and outside. Optionally, using an open source python package for calculating positional relationships, the positional relationship between entities is accurately calculated, and finally a structured information of each entity is obtained. Based on such information, the annotation of each information ground truth is realized.

[0058] Regarding the above step S102, in an optional embodiment, the image information is converted into language information, and the convolutional architecture caffee of the computer vision model for fast feature embedding is used as the information detection and extraction model. The language information and the preset known information are overlapped to obtain the calculation result, and the entity information is supplemented according to the calculation result to obtain the supplemented language information. For example, according to the color feature, the color information is attached to the corresponding visual entity language description in the form of a text description, and the inferred state information is added to the language description of the visual entity in the form of a text description. This optional embodiment implements a method for converting visual information into language information, and supplements the color and state information through the Caffe model, thereby improving the description accuracy and application value of the visual information. The above supplemented language information is converted into subject-predicate-object SPO triple information to construct the knowledge graph. In the field of natural language processing and information extraction, the full Chinese name of SPO triple information usually refers to the "Subject-Predicate-Object" triple. This concept comes from linguistics and is used to describe the basic syntactic structure in a sentence, where: Subject: The subject in a sentence is usually a person or thing that performs an action or is in a certain state. Predicate: The action or state in a sentence, connecting the subject and the object, describing the action performed by the subject or the state it is in. Object: The recipient of the action or state in a sentence, that is, the object being acted upon. In information extraction tasks, SPO triples are used to extract structured information from text and convert unstructured text data into a structured data form that is easy to process. For example, in the construction of knowledge graphs, SPO triples are the basic units that constitute facts.

[0059] In order to improve the efficiency and accuracy of information conversion and natural language generation, in an optional embodiment, the subject-verb-object SPO triple sample data of the input entity is collected, and the data volume can be 1k. The Qwen-1.5-7b pre-trained model is used as the basic model, and the parameter efficient fine-tuning technology LoRA technology is used to fine-tune the basic model; wherein, the training parameters include a learning rate (lr) of 1e-3 and a training round (epoch) of 10, which realizes efficient fine-tuning of the model at a lower learning rate (lr=1e-3) and a limited training round (epoch=10), reducing the training time and computing resource requirements. The basic model is trained with the subject-verb-object SPO triple sample data to obtain a trained model, and the trained model is used to convert the subject-verb-object SPO triple information into information with a fluent and rich natural language description. Through this optional embodiment, in order to prune information, reduce complexity, and more accurately locate the entities currently required, the Qwen-1.5-7b pre-trained model is used in combination with the LoRA fine-tuning technology. The trained model can convert the subject-verb-object SPO triple information into a fluent and rich natural language description, thereby improving the model's advanced cognitive capabilities, including reasoning, comparison, and description.

[0060] LoRA (Low-Rank Adaptation of Large Language Models) is a parameter-efficient fine-tuning technique that allows us to approximate the effect of full parameter fine-tuning by introducing additional low-rank matrices without changing the original model parameters. The core idea of ​​this method is to use low-rank matrices to represent the changes in model parameters, thereby significantly reducing the number of parameters that need to be trained and reducing computational and memory costs. Specifically, LoRA simulates the amount of parameter updates by adding a bypass next to specific layers of the model (such as the query, key, value, and output projection matrices in the self-attention layer of the Transformer). This bypass consists of two small matrices whose inner products are consistent with the input and output shapes of the original model. The key to this method is that the number of newly added parameters is much smaller than the number of parameters of the original model, so the cost of fine-tuning can be significantly reduced. In practical applications, LoRA can maintain performance similar to traditional fine-tuning methods while significantly reducing computational costs. For example, in natural language processing tasks, the performance of models fine-tuned using LoRA only dropped by less than 1% when 90% of parameters were reduced. In addition, LoRA can also be combined with other NLP technologies to further improve model performance, such as combining with knowledge graphs to use external knowledge to guide the fine-tuning process. The training process of LoRA is similar to traditional model training, except that the LoRA layer is integrated into the model and only the parameters of these LoRA layers are updated during training. In this way, the weights of other parts of the original model remain unchanged, making the training process more efficient. In general, LoRA is an effective large model fine-tuning technology that achieves rapid model adaptation and performance improvement by introducing a small number of parameters, while significantly reducing the computational and storage costs of fine-tuning.

[0061] Regarding the above-mentioned step S104, in an optional embodiment, the multimodal artificial intelligence model GPT-4o is called to generate descriptive text for the input sub-region image. By calling the multimodal artificial intelligence model GPT-4o, the content of the sub-region image can be understood more accurately and descriptive text can be generated, thereby improving the overall image parsing ability. The descriptive text is combined with the information description of the central entity and input into the language model LLM to generate single-hop question-answer pairs and multi-hop question-answer pairs; wherein the single-hop question-answer pair refers to a question and answer directly based on the picture description, and the multi-hop question-answer pair refers to a question and answer that requires reasoning or additional information processing. Combining the image descriptive text with the information description of the central entity can more comprehensively integrate the image and text information and provide rich context for generating question-answer pairs. The technical solution of this optional embodiment not only improves the efficiency and accuracy of multimodal data processing by combining the GPT-4o model and LLM, but also enhances the reasoning and question-answer generation capabilities of the model, especially when dealing with multi-hop question-answering tasks that require complex reasoning.

[0062] The following is an illustration with reference to specific embodiments.

[0063] Perception qa pairs perception question answering dataset construction: * 7 keywords (i.e. one-hop operators) [Position] Why--> Position is the best way to locate an object in space. For example, "the red apple on the table" and "the red apple in the refrigerator" can help humans accurately locate two different types of apples.

[0064] [Color] Why--> Color is a description of the properties of an object. It can be used to distinguish identical objects. Such visual signals can also be used to find objects. For example, "the earphones on the table near the mouse are white", "the earphones on the right side of the keyboard on the table are purple", so we can distinguish the colors of the two earphones on the table.

[0065] [Quantity] Why--> Quantity is a description of the spatial state of an object. If it does not exist, it is 0. If it exists, it may be one or more, such as "there are two toys, Lego and Rubik's Cube, on the toy table."

[0066] [Volume] Why--> Volume is a description of the size of an object. It can distinguish objects or characterize objects. For example, "the volume of the apple on the table is 5", "the volume of the apple in the refrigerator is 8". Therefore, if you want to eat a big apple, you should go to the refrigerator.

[0067] [Category] Why--> Category is a description of the properties of an object, which can distinguish different objects, such as "apple is a kind of fruit", "apple-shaped cup is a kind of water cup, and water cup is a kind of container".

[0068] [Function] Why--> Function is a description of the function of an object, which can distinguish different objects, such as "there are apples on the coffee table in the living room that can quench thirst", "there are books in the study that can enrich knowledge".

[0069] [Composition relationship] Why--> Composition is a description of the existence state of an object, showing that some objects can be composed of other objects, such as "a cabinet is composed of drawers and cabinet frames", "a lock is part of a door".

[0070] The basic perception data set includes the following issues: existence issues, positional relationships, attribute relationships, functional attributes, size relationships and composition relationships, and counting capabilities.

[0071] Construction of reasoning qa pairs cognitive question answering dataset: Cognitive data includes: combination, nesting, comparison, and reasoning question and answer for one-hop questions.

[0072] In this embodiment, a device for constructing a multimodal perception and cognitive data set is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0073] Figure 2 is a structural block diagram of a device for constructing a multimodal perception and cognitive data set according to an embodiment of the present invention, such as Figure 2 As shown, the device comprises: The acquisition module 21 is used to acquire image information; wherein the image information includes the picture captured by the camera of the intelligent body in real time; A first construction module 22 is used to construct a knowledge graph based on the image information; wherein the knowledge graph includes entity information and positional relationships between entities; A segmentation module 23, used for segmenting the image information into non-overlapping sub-region images; The second construction module 24 is used to describe the image based on the sub-region image and the knowledge graph, generate a natural language text describing the image content, and construct a multimodal perception and cognition dataset based on the natural language text.

[0074] Figure 3 is a structural block diagram of a first building block according to an embodiment of the present invention, such as Figure 3 As shown, the first building block 22 includes: A conversion unit 221, used to convert the image information into language information; The calculation unit 222 is used to use the convolution architecture caffee of the computer vision model for fast feature embedding as the information detection and extraction model, calculate the overlap between the language information and the preset known information, and obtain a calculation result; A supplementing unit 223, used to supplement the entity information according to the calculation result to obtain supplemented language information; The construction unit 224 is used to convert the supplemented language information into subject-verb-object (SPO) triple information to construct the knowledge graph.

[0075] Figure 4 is another structural block diagram of a device for constructing a multimodal perception and cognitive data set according to an embodiment of the present invention, such as Figure 4 As shown, the device also includes: A collection module 41 is used to collect subject-verb-object SPO triple sample data of the input entity; Selection module 42 is used to adopt the Qwen-1.5-7b pre-trained model as a basic model; A fine-tuning module 43 is used to fine-tune the basic model using the parameter efficient fine-tuning technology LoRA technology; wherein the training parameters include a learning rate (lr) of 1e-3 and a training round (epoch) of 10; A training module 44 is used to train the basic model using the subject-verb-object SPO triple sample data to obtain a trained model; The conversion module 45 is used to convert the subject-verb-object (SPO) triple information into information with fluent and rich natural language description through the trained model.

[0076] Figure 5 is a structural block diagram of a second building block according to an embodiment of the present invention, such as Figure 5 As shown, the second building block 24 includes: The first generating unit 241 is used to call the multimodal artificial intelligence model GPT-4o to generate descriptive text for the input sub-region image; The second generation unit 242 is used to combine the descriptive text with the information description of the central entity and input it into the language model LLM to generate single-hop question-answer pairs and multi-hop question-answer pairs; wherein the single-hop question-answer pair refers to a question and answer directly based on the image description, and the multi-hop question-answer pair refers to a question and answer that requires reasoning or additional information processing.

[0077] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0078] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0079] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0080] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0081] refer to Figure 6 , a block diagram of an electronic device that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0082] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0083] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0084] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the music data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform a method according to an embodiment of the present invention in any other appropriate manner (e.g., by means of firmware).

[0085] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0087] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0089] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0090] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0091] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for constructing a multimodal perception and cognitive dataset, characterized in that: The method comprises: Acquire image information; wherein the image information includes a picture captured in real time by the camera of the intelligent agent itself; Constructing a knowledge graph based on the image information; wherein the knowledge graph includes entity information and positional relationships between entities; Dividing the image information into non-overlapping sub-region images; An image description is performed based on the sub-region image and the knowledge graph, a natural language text describing the image content is generated, and a multimodal perception and cognition dataset is constructed based on the natural language text.

2. The method for constructing a multimodal perception and cognitive dataset according to claim 1, characterized in that: Dividing the image information into non-overlapping sub-region images comprises the following steps: S1: converting the image information into a pixel feature matrix, wherein each pixel point is regarded as a sample, and normalizing the pixel features to obtain a processed image; S2: Determine the number of clusters according to the elbow rule and the silhouette coefficient; where the number of clusters represents the number of sub-region images; S3: Randomly select the initial cluster center through the K-means clustering algorithm; S4: For each pixel in the image, calculate its Euclidean distance to each initial cluster center, and assign it to the cluster represented by the nearest initial cluster center; S5: Calculate the mean of all points in each cluster by using the K-means clustering algorithm, and recalculate the cluster center of each cluster according to the mean; Steps S3, S4, and S5 are repeatedly performed until the change value of the cluster center of each cluster is less than a predetermined threshold, or the number of iterations reaches a preset number of iterations, or the change of samples within the cluster is less than a preset change amount.

3. The method for constructing a multimodal perception and cognitive dataset according to claim 1, characterized in that: Before constructing the knowledge graph according to the image information, the method further includes: Obtaining coarse-grained information of the image information; wherein granularity is used to describe the granularity level of image processing or analysis, and the coarse-grained information is obtained by performing global processing on the image, focusing on the characteristics of the overall image; The labels of the coarse-grained information are refined through a visual target detection model to obtain fine-grained entity information and the positional relationship between fine-grained entities; wherein fine-grained means local subclass division and recognition of the image; the positional relationship between fine-grained entities includes up, down, left, right, front, back, inside, and outside.

4. The method for constructing a multimodal perception and cognitive dataset according to claim 1, characterized in that: Constructing a knowledge graph according to the image information includes: converting the image information into language information; Using the convolutional architecture caffee for fast feature embedding of computer vision models as an information detection and extraction model, the language information is overlapped with the preset known information to obtain a calculation result; Supplementing the entity information according to the calculation result to obtain supplemented language information; The supplemented language information is converted into subject-verb-object (SPO) triple information to construct the knowledge graph.

5. The method for constructing a multimodal perception and cognitive dataset according to claim 4, characterized in that: After constructing the knowledge graph according to the image information, the method further includes: Collect the subject-verb-object (SPO) triple sample data of the input entity; The Qwen-1.5-7b pre-trained model is used as the basic model; The basic model is fine-tuned using the parameter efficient fine-tuning technology LoRA technology; wherein the training parameters include a learning rate (lr) of 1e-3 and a training round (epoch) of 10; Using the subject-verb-object SPO triple sample data to train the basic model to obtain a trained model; The trained model is used to transform subject-verb-object (SPO) triple information into information with fluent and rich natural language description.

6. The method for constructing a multimodal perception and cognitive dataset according to any one of claims 1 to 5, characterized in that: Performing image description according to the sub-region image and the knowledge graph, generating natural language text describing the image content, and constructing a multimodal perception and cognition dataset according to the natural language text includes: Call the multimodal artificial intelligence model GPT-4o to generate descriptive text for the input sub-region image; The descriptive text is combined with the information description of the central entity and input into the language model LLM to generate single-hop question-answer pairs and multi-hop question-answer pairs; wherein the single-hop question-answer pairs refer to questions and answers directly based on the image description, while the multi-hop question-answer pairs refer to questions and answers that require reasoning or additional information processing.

7. A device for constructing a multimodal perception and cognitive data set, characterized in that: The device comprises: An acquisition module, used to acquire image information; wherein the image information includes a picture captured in real time by the camera of the intelligent agent itself; A first construction module is used to construct a knowledge graph based on the image information; wherein the knowledge graph includes entity information and positional relationships between entities; A segmentation module, used for segmenting the image information into non-overlapping sub-region images; The second construction module is used to describe the image based on the sub-region image and the knowledge graph, generate natural language text describing the image content, and construct a multimodal perception and cognition dataset based on the natural language text.

8. The device for constructing a multimodal perception and cognitive data set according to claim 7, characterized in that: The first building block comprises: A conversion unit, used for converting the image information into language information; A computing unit, used for using a convolutional architecture caffee for fast feature embedding of a computer vision model as an information detection and extraction model, calculating the overlap between the language information and preset known information, and obtaining a calculation result; A supplementing unit, used to supplement the entity information according to the calculation result to obtain supplemented language information; A construction unit is used to convert the supplemented language information into subject-verb-object (SPO) triple information to construct the knowledge graph.

9. The device for constructing a multimodal perception and cognitive dataset according to claim 8, characterized in that: The device also includes: A collection module is used to collect subject-verb-object SPO triple sample data of input entities; Select the module to use the Qwen-1.5-7b pre-trained model as the base model; A fine-tuning module, used to fine-tune the basic model using the parameter efficient fine-tuning technology LoRA technology; wherein the training parameters include a learning rate (lr) of 1e-3 and a training round (epoch) of 10; A training module, used to train the basic model using the subject-verb-object SPO triple sample data to obtain a trained model; The conversion module is used to convert the subject-verb-object (SPO) triple information into information with fluent and rich natural language description through the trained model.

10. The device for constructing a multimodal perception and cognitive dataset according to any one of claims 7 to 9, characterized in that: The second building block comprises: A first generating unit is used to call the multimodal artificial intelligence model GPT-4o to generate descriptive text for the input sub-region image; The second generation unit is used to combine the descriptive text with the information description of the central entity and input it into the language model LLM to generate single-hop question-answer pairs and multi-hop question-answer pairs; wherein the single-hop question-answer pairs refer to questions and answers directly based on the image description, and the multi-hop question-answer pairs refer to questions and answers that require reasoning or additional information processing.

11. An electronic device comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

13. A computer program product, characterized in that The computer program product comprises instructions which, when executed, cause a computer to perform the method according to any one of claims 1 to 6.

Images