Data acquisition system and method based on visual big language model
Through the visual large language model combined with the cross-modal attention mechanism, the feature fusion of image and text data is solved, and the problems of low data acquisition efficiency and poor quality in the existing technology are realized, and automated and efficient multimodal data acquisition is achieved.
Patent Information
- Application Number
- CN202411407639.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing data acquisition methods rely on a large number of manual interventions, are inefficient and have poor data quality, especially in multimodal data fusion and complex environments, making it difficult for traditional methods to achieve automated and efficient data acquisition.
Using a data acquisition system based on a large visual language model, combined with computer vision and natural language processing technology, the image and text data are extracted and fused through a cross-modal attention mechanism to generate multimodal feature vectors, reduce manual intervention, and improve Efficiency and accuracy of data acquisition.
The data acquisition process is automated, the efficiency and accuracy of data acquisition are significantly improved, manual intervention is reduced, and it is highly adaptable, and is suitable for multimodal data processing.
Smart Images

Figure CN118916841A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of artificial intelligence and machine learning, and in particular, relates to a data acquisition system and method based on a visual large language model. Background Art
[0002] Driven by the development of modern science and technology, data has become a key resource to promote the progress of artificial intelligence and machine learning. As the core technology of data acquisition, data acquisition system is widely used in many fields, such as autonomous driving, intelligent monitoring, medical diagnosis and smart home. The existing data acquisition methods mainly rely on sensors and manual annotation, including the following types: (1) Sensor-based data acquisition system: Use various sensors (such as cameras, microphones, temperature sensors, etc.) to collect environmental data. This type of system is usually used in real-time monitoring, environmental monitoring, and automatic control. The sensor-based data acquisition system collects raw data, which requires subsequent complex processing and labeling; (2) Manual data labeling system: In many AI applications, especially supervised learning tasks, manually labeled data is indispensable. Manual data labeling requires a lot of human resources, the labeling process is time-consuming and costly, and there are problems of human error and inconsistent data quality; (3) Semi-automatic data collection system: A system that combines automated tools with human intervention, using preliminary computer vision or natural language processing technology for preliminary data processing, followed by manual verification and correction. Although this improves efficiency to a certain extent, it still cannot completely avoid human involvement.
[0003] With the development of big data and AI technology, the demand and complexity of data collection are increasing. Traditional data collection methods have many shortcomings in terms of efficiency and accuracy, such as low efficiency and poor data quality. In addition, manually annotated data is prone to noise and inconsistency, which affects the training effect of the model. In addition, the data collected by sensors may be disturbed in complex environments, resulting in reduced data quality. Traditional data acquisition systems perform poorly when dealing with complex and dynamic scenes. For example, in terms of the fusion and understanding of multimodal data (images, videos, text, etc.), existing technologies lack effective solutions. Traditional methods rely on a lot of manual intervention, and data collection is slow and costly. This is particularly prominent in AI training tasks that require large-scale data.
[0004] Therefore, technicians in this field are committed to developing a data acquisition system and method based on a visual large language model. Summary of the invention
[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to realize the automation of the data collection process, reduce the dependence on manual intervention, and improve the efficiency and accuracy of data collection.
[0006] In recent years, the development of visual big language models has brought new opportunities to data collection systems. Visual big language models combine the technical advantages of computer vision and natural language processing to simultaneously process and understand visual data and language data.
[0007] To achieve the above objectives, this paper analyzes the shortcomings of the existing data acquisition system in terms of efficiency, data quality and adaptability. Combining the technical advantages of computer vision and natural language processing, this paper simultaneously processes and understands visual data and language data, extracts and analyzes features of image data and video data, combines image data and language data, and fuses image feature vectors and text feature vectors through a cross-modal attention mechanism to generate a multimodal feature vector, thereby realizing multimodal data fusion.
[0008] In one embodiment of the present invention, a data collection system based on a visual large language model is provided, comprising: The data input module receives input data, performs data preprocessing, and then sends it to the visual language model processing module; The visual large language model processing module resides in the visual large language model. In response to the data sent by the data input module, based on the Transformer technology, it uses the cross-modal attention mechanism to comprehensively understand and process, generates a multimodal feature vector, and sends it to the data storage module; A data storage module performs quality control on the consistency and accuracy of multimodal feature vectors and stores them; Data output module, which obtains multimodal feature vectors, performs format conversion and outputs them externally; The data input module, the visual large language model processing module, the data storage module and the data output module are sequentially connected in communication; The data input module receives input data and sends it to the visual large language model processing module after preprocessing. The visual large language model processing module uses the visual large language model, based on Transformer technology, and uses the cross-modal attention mechanism for comprehensive understanding and processing to generate a multimodal feature vector, which is sent to the data storage module. The data storage module performs quality control on the consistency and accuracy of the multimodal feature vector, and stores and manages it. The data output module obtains the data stored in the data storage module, converts the format, and outputs it.
[0009] Optionally, in the data acquisition system based on the visual large language model in the above embodiment, the input data includes image data, video data and text data.
[0010] Optionally, in the data acquisition system based on the visual large language model in any of the above embodiments, sources of input data include sensor acquisition and data import.
[0011] Optionally, in the data acquisition system based on the visual large language model in any of the above embodiments, the data input module includes an image sensor and a voice sensor.
[0012] Furthermore, in the data acquisition system based on the visual large language model in the above embodiment, the image sensor includes a camera and a video camera.
[0013] Furthermore, in the data acquisition system based on the visual large language model in the above embodiment, the speech sensor includes a microphone.
[0014] Furthermore, in the data acquisition system based on the visual large language model in the above embodiment, data preprocessing includes image enhancement, text denoising, data normalization, data compression and data decompression.
[0015] Optionally, in the data acquisition system based on the visual big language model in any of the above embodiments, the visual big language model processing module includes: The visual data processing unit uses the visual Transformer technology to extract and analyze the features of the image data and the video data, and generates an image feature vector including the visual features in the image data and the video data; The language data processing unit uses the text Transformer technology to process and understand the text data and generate text feature vectors; The multimodal data fusion unit, through the cross-modal attention mechanism, fuses the image feature vector and the text feature vector to generate a multimodal feature vector containing both visual and text information.
[0016] Furthermore, in the data collection system based on the visual large language model in the above embodiment, processing and understanding text data includes: Preprocess text data to remove irrelevant information and improve data quality, including removing spaces, line breaks, special characters, and stop words; Encoding: Use text Transformer technology to encode the preprocessed text data; Text feature vector generation, extracts semantic features from text and generates text feature vectors, including semantic information in the text.
[0017] Furthermore, in the data collection system based on the visual large language model in the above embodiment, the semantic information includes keywords, topics, and sentence structures.
[0018] Based on any of the above embodiments, in another embodiment of the present invention, a data collection method based on a visual large language model is provided, comprising the following steps: S100, data preprocessing, collecting input data and performing data preprocessing; S200, feature extraction, based on the visual language model, uses Transformer technology to extract features from the preprocessed data to generate image feature vectors and text feature vectors; S300, multimodal information fusion, through the cross-modal attention mechanism, the image feature vector and the text feature vector are fused to generate a multimodal feature vector, which includes visual information and text information; S400, data storage, performing quality control on consistency and accuracy of multimodal feature vectors and storing them; S500, data output, obtaining the stored multimodal feature vector, performing format conversion and outputting it.
[0019] Optionally, in the data collection method based on the visual large language model in the above embodiment, step S100 includes: S110, data collection, receiving input data; S120: Data preprocessing: preprocessing the input data.
[0020] Furthermore, in the data collection method based on the visual large language model in the above embodiment, the input data includes image data, video data and text data.
[0021] Optionally, in the data collection method based on the visual large language model in any of the above embodiments, the source of input data includes sensor collection and data import.
[0022] Furthermore, in the data collection method based on the visual large language model in the above embodiment, step S120 includes: S121, Image enhancement, enhancing the visual quality of images through basic image processing methods; S122, text denoising, performing denoising processing on text data to improve the quality of the text data; S123, data normalization, normalizing the image data to ensure that the data format is consistent for subsequent processing; S124, data compression, compressing the image data and the text data into a unified feature vector space at the same time; S125, data decompression, decompression from the feature vector space to obtain image data and text data.
[0023] Furthermore, in the data acquisition method based on the visual large language model in the above embodiment, the basic image processing method includes brightness adjustment, contrast adjustment, sharpening, rotation, translation, and scaling.
[0024] Optionally, in the data collection method based on the visual large language model in any of the above embodiments, step S200 includes: S210, extracting image feature vectors, using visual Transformer technology to extract and analyze features of image data and video data in the preprocessed data, and generating image feature vectors including visual features in the image data and video data; S220, text feature vector extraction, using text Transformer technology to process and understand the text data in the preprocessed data to generate a text feature vector.
[0025] Optionally, in the data collection method based on the visual large language model in any of the above embodiments, step S400 includes: S410, quality control, compares the multimodal feature vectors of image data and text data through cosine similarity to ensure consistency; S420: storing the multimodal feature vector that has undergone quality control.
[0026] Optionally, in the data collection method based on the visual large language model in any of the above embodiments, step S500 includes: S510, obtaining a multimodal feature vector, and obtaining a stored multimodal feature vector through a visual large language model; S520, format conversion, mapping the multimodal feature vector to image data and text data; S530: Output: output the mapped image data and text data.
[0027] The present invention combines the technologies of computer vision and natural language processing to simultaneously process and understand visual data and language data. By combining the visual large language model and multimodal data processing technology, the present invention realizes the intelligence and automation of the data acquisition system, reduces the dependence on human intervention, and significantly improves the efficiency, accuracy and adaptability of data acquisition.
[0028] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a structural schematic diagram illustrating a data acquisition system based on a visual large language model according to an exemplary embodiment; Figure 2 is a flowchart illustrating a data collection method based on a visual large language model according to an exemplary embodiment. DETAILED DESCRIPTION
[0030] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.
[0031] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is schematically exaggerated in some places in the drawings.
[0032] The inventors have designed a data collection system for a visual large language model, such as Figure 1 As shown, including: The data input module receives input data, performs data preprocessing, and then sends it to the visual large language model processing module; the input data includes image data, video data and text data, and the source of the input data includes sensor acquisition and data import; the data input module includes image sensors and voice sensors, image sensors include cameras and video cameras, and voice sensors include microphones; data preprocessing includes image enhancement, denoising, data normalization, data compression and data decompression.
[0033] The visual large language model processing module resides in the visual large language model. In response to the data sent by the data input module, based on the Transformer technology, it uses the cross-modal attention mechanism to comprehensively understand and process, generates a multimodal feature vector, and sends it to the data storage module. The visual large language model processing module includes: The visual data processing unit uses the visual Transformer technology to extract and analyze the features of the image data and the video data, and generates an image feature vector including the visual features in the image data and the video data.
[0034] The language data processing unit uses the text Transformer technology to process and understand the text data and generate text feature vectors. The processing and understanding of text data includes: Preprocess text data to remove irrelevant information and improve data quality, including removing spaces, line breaks, special characters, and stop words; Encoding: Use text Transformer technology to encode the preprocessed text data; Text feature vector generation, extracting semantic features from text and generating text feature vectors, including semantic information in the text. Semantic information includes keywords, topics, and sentence structures.
[0035] The multimodal data fusion unit, through the cross-modal attention mechanism, fuses the image feature vector and the text feature vector to generate a multimodal feature vector containing both visual and text information.
[0036] The data storage module performs quality control on the consistency and accuracy of the multimodal feature vectors and stores them.
[0037] The data output module obtains multimodal feature vectors, performs format conversion and outputs them externally. The data output module supports output in multiple formats, including CSV, JSON, and JPEG.
[0038] The data input module, the visual large language model processing module, the data storage module and the data output module are communicatively connected in sequence.
[0039] The data input module receives input data and sends it to the visual large language model processing module after preprocessing. The visual large language model processing module uses the visual large language model, based on Transformer technology, and uses the cross-modal attention mechanism for comprehensive understanding and processing to generate a multimodal feature vector, which is sent to the data storage module. The data storage module performs quality control on the consistency and accuracy of the multimodal feature vector, and stores and manages it. The data output module obtains the data stored in the data storage module, converts the format, and outputs it.
[0040] Based on any of the above embodiments, in another embodiment of the present invention, a data collection method based on a visual large language model is provided, comprising the following steps: S100, data preprocessing, collecting data and performing data preprocessing, including: S110, data collection, receiving input data, the source of the input data includes sensor collection and data import, the input data includes image data, video data and text data; S120, data preprocessing, preprocessing the input data, including: S121, Image enhancement, enhances the visual quality of images through basic image processing methods, including brightness adjustment, contrast adjustment, sharpening, rotation, translation, and scaling; S122, text denoising, performing denoising processing on text data to improve the quality of the text data; S123, data normalization, normalizing the image data to ensure that the data format is consistent for subsequent processing; S124, data compression, compressing the image data and the text data into a unified feature vector space at the same time; S125, data decompression, decompression from the feature vector space to obtain image data and text data.
[0041] S200, feature extraction, based on the visual language model, uses Transformer technology to extract features from preprocessed data and generate image feature vectors and text feature vectors, including: S210, extracting image feature vectors, using visual Transformer technology to extract and analyze features of image data and video data in the preprocessed data, and generating image feature vectors including visual features in the image data and video data; S220, text feature vector extraction, using text Transformer technology to process and understand the text data in the preprocessed data to generate a text feature vector.
[0042] S300, multimodal information fusion, fuses the image feature vector and the text feature vector through a cross-modal attention mechanism to generate a multimodal feature vector containing visual information and text information.
[0043] S400, data storage, quality control of the consistency and accuracy of the multimodal feature vectors and storage, including: S410, quality control, compares the multimodal feature vectors of image data and text data through cosine similarity to ensure consistency; S420: storing the multimodal feature vector that has undergone quality control.
[0044] S500, data output, obtaining the stored multi-modal feature vector, performing format conversion and outputting, including: S510, obtaining a multimodal feature vector, and obtaining a stored multimodal feature vector through a visual large language model; S520, format conversion, mapping the multimodal feature vector to image data and text data; S530: Output: output the mapped image data and text data.
[0045] In order to verify the technical effect of the present invention, an experiment was conducted. The verification environment is as follows: (1) Hardware •GPU: NVIDIA A100 40GB.
[0046] •CPU: Intel Xeon Platinum 8275CL.
[0047] •Memory: 512 GB RAM.
[0048] • Storage: 20 TB SSD.
[0049] •Sensor equipment: high-resolution cameras, 4K video cameras, omnidirectional microphones.
[0050] (2) Software •Operating system: Ubuntu 20.04 LTS.
[0051] •Deep learning framework: PyTorch 2.0.
[0052] •Data processing tools: OpenCV 4.5.5, NLTK 3.6.
[0053] •Database: PostgreSQL 13.
[0054] •Programming language: Python 3.9.
[0055] Based on the data collection system environment of a visual large language model, experiments were conducted according to a data collection method of a visual large language model. The experimental results were evaluated from the following aspects and compared with the existing technology: 1. Test result evaluation indicators •Data processing time, the total time from data input to final data output of the data acquisition system, in seconds; • Feature extraction accuracy, by comparing the cosine similarity between the original data and the extracted feature vectors to evaluate the accuracy of feature extraction; •Multimodal fusion effect, evaluate the performance of the fused multimodal feature vector in downstream tasks (such as image classification and text generation), including accuracy and F1 score; • Storage efficiency, which measures the storage size of each feature vector and its comparison with the size of the original data.
[0056] 2. Test results The data processing of existing technologies uses classic network architectures such as convolutional neural networks to process visual information, and uses classic natural language technologies such as word embedding to capture text semantic relationships. It has limited ability to understand context, and it is difficult to handle complex language representations, and it is unable to fully and accurately capture the deep associations between vision and text. The fusion of visual and language features usually relies on linear concatenation or simple weighted average, that is, concatenating or linearly combining visual feature vectors and text feature vectors. Although this method can integrate information from different modalities, it lacks modeling of deep relationships between modalities and cannot fully play a role in complex multimodal tasks. As a result, the performance of existing technologies in downstream tasks is relatively poor. In terms of storage, existing technologies usually do not perform feature compression or optimization, the storage size of each feature vector is large, the visual features are often of high dimensionality, and no further dimensionality reduction processing has been performed, resulting in lengthy and redundant feature vectors.
[0057] The following table compares the evaluation indicators of the present invention and the prior art: It can be seen that the present invention is much better than the prior art in terms of data processing time, feature extraction accuracy, multimodal fusion accuracy and feature vector storage efficiency. The present invention realizes the automation of the data collection process, reduces the dependence on manual intervention, and improves the efficiency and accuracy of data collection.
[0058] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A data acquisition system based on a visual large language model, characterized in that: include: The data input module receives input data, performs data preprocessing, and then sends it to the visual language model processing module; A visual large language model processing module, in which a visual large language model resides, responds to the data sent by the data input module, uses a cross-modal attention mechanism to comprehensively understand and process the data based on the Transformer technology, generates a multimodal feature vector, and sends it to the data storage module; A data storage module performs quality control on the consistency and accuracy of the multimodal feature vector and stores the same; A data output module obtains the multimodal feature vector, performs format conversion and outputs it externally; The data input module, the visual large language model processing module, the data storage module and the data output module are communicatively connected in sequence.
2. The data acquisition system based on the visual large language model as claimed in claim 1, characterized in that: The input data includes image data, video data and text data.
3. The data acquisition system based on the visual large language model as claimed in claim 1, characterized in that: The sources of the input data include sensor acquisition and data import.
4. The data acquisition system based on the visual large language model as claimed in claim 1, characterized in that: The data preprocessing includes image enhancement, text denoising, data normalization, data compression and data decompression.
5. The data acquisition system based on the visual large language model as claimed in claim 1, characterized in that: The visual large language model processing module includes: The visual data processing unit uses the visual Transformer technology to extract and analyze the features of the image data and the video data, and generates an image feature vector including the visual features in the image data and the video data; The language data processing unit uses the text Transformer technology to process and understand the text data and generate text feature vectors; The multimodal data fusion unit, through the cross-modal attention mechanism, fuses the image feature vector and the text feature vector to generate a multimodal feature vector containing both visual and text information.
6. The data acquisition system based on the visual large language model as claimed in claim 5, characterized in that: The processing and understanding of text data includes: Preprocess text data to remove irrelevant information and improve data quality, including removing spaces, line breaks, special characters, and stop words; Encoding: Use text Transformer technology to encode the preprocessed text data; Text feature vector generation, extracts semantic features from text and generates text feature vectors, including semantic information in the text.
7. A data collection method based on a visual large language model, using the data collection system based on a visual large language model as described in any one of claims 1 to 6, characterized in that: The steps include: S100, data preprocessing, collecting data and performing data preprocessing; S200, feature extraction, based on the visual language model, uses Transformer technology to extract features from the preprocessed data to generate image feature vectors and text feature vectors; S300, multimodal information fusion, fusing the image feature vector and the text feature vector through a cross-modal attention mechanism to generate a multimodal feature vector including visual information and text information; S400, data storage, performing quality control on the consistency and accuracy of the multimodal feature vector, and storing the data; S500, data output, obtaining the stored multimodal feature vector, performing format conversion and outputting it.
8. The data collection method based on the visual large language model according to claim 7, characterized in that: The step S100 includes: S110, data collection, receiving input data; S120: Data preprocessing: preprocessing the input data.
9. The data collection method based on the visual large language model according to claim 7 or 8, characterized in that: The step S200 includes: S210, extracting image feature vectors, using visual Transformer technology to extract and analyze features of image data and video data in the preprocessed data, and generating image feature vectors including visual features in the image data and video data; S220, text feature vector extraction, using text Transformer technology to process and understand the text data in the preprocessed data to generate a text feature vector.
10. The data collection method based on the visual large language model according to claim 9, characterized in that: The step S500 includes: S510, obtaining a multimodal feature vector, and obtaining a stored multimodal feature vector through the visual large language model; S520, format conversion, mapping the multimodal feature vector to image data and text data; S530: Outputting the mapped image data and the text data.
Citation Information
Patent Citations
Multi-modal text page classification method based on decoupling feature guidance
CN115761757A
Substation visual scene description method based on visual language model
CN117786588A
Visually controllable natural language generation method and device, electronic equipment and medium
CN117851541A
