Data processing method and device

Abstract

The invention discloses a data processing method and device. A data processing method may include the following steps. A processing device of a data processing device obtains structured data and unstructured data corresponding to each piece of input data according to a plurality of pieces of input data. And integrating the structured data and the unstructured data through the processing device so as to generate integrated data corresponding to each piece of input data. Through the processing device, the integrated data is stored in a retrieval enhancement generation database so as to be provided for a large language model to use.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a data processing technique, and in particular, to a data processing technique for generating image semantic data. BACKGROUND

[0002] With the development of technology, the application of artificial intelligence (AI) is also becoming more and more widespread. Large language models (LLM) are a kind of artificial intelligence program that can recognize the user's query and prompt to generate a response. Retrieval augmented generation (RAG) is a method that combines retrieval and generation techniques. RAG can first retrieve relevant information from an external database, and then use this information as auxiliary data for LLM system reference.

[0003] However, when the LLM system needs to process multi-modal data (such as unstructured data and structured data), the LLM system needs to retrieve different modal data from different databases and then integrate the retrieved data. Therefore, the LLM system will consume a lot of computing and retrieval time.

[0004] Therefore, how to more efficiently retrieve data so that the LLM system can generate a response more quickly and accurately is a topic worthy of study. SUMMARY

[0005] In view of the problems of the prior art, embodiments of the present disclosure provide a data processing method and device.

[0006] According to an embodiment of the present disclosure, a data processing method is provided. The data processing method can include the following steps. According to a plurality of input data, a structured data and an unstructured data corresponding to each input data are obtained by a processing device of a data processing device. The structured data and the unstructured data are integrated by the processing device to generate an integrated data corresponding to each input data. The integrated data is stored in a retrieval augmented generation (RAG) database by the processing device to provide a large language model (LLM) for use.

[0007] According to an embodiment of the present application, a data processing apparatus is provided. The data processing apparatus can include a processing device and a retrieval augmentation generation database. The processing device can obtain a structured data and an unstructured data corresponding to each of a plurality of input data according to the plurality of input data, and integrate the structured data and the unstructured data to generate an integrated data corresponding to each of the plurality of input data. The retrieval augmentation generation database can store the integrated data to provide a large language model.

[0008] Other additional features and advantages of the present application will be apparent from the following detailed description of the application, taken in conjunction with the accompanying drawings of which: BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 A block diagram of a data processing apparatus of the present application;

[0010] Figure 2 A schematic diagram of generating an image semantic data according to an embodiment of the present application;

[0011] Figure 3 A schematic diagram of subject grouping according to an embodiment of the present application;

[0012] Figure 4 A schematic diagram of word frequency of each subject according to an embodiment of the present application;

[0013] Figure 5 A schematic diagram of a program of a large language model according to an embodiment of the present application;

[0014] Figure 6 A flowchart of a data processing method according to an embodiment of the present application.

[0015] LIST OF ABBREVIATIONS

[0016] 100: data processing apparatus

[0017] 110: processing device

[0018] 120: storage device

[0019] 130: input / output device

[0020] 140: image capturing device

[0021] 210: input image

[0022] 220, 230: block

[0023] 510: RAG program

[0024] 520: LLM program

[0025] S610~S630: Steps Detailed Implementation

[0026] This section describes preferred embodiments of the invention and is intended to illustrate the spirit of the invention rather than to limit its scope of protection. The scope of protection of the invention shall be defined by the appended claims.

[0027] Figure 1 This is a block diagram showing a data processing apparatus 100 according to the present invention. (Example) Figure 1 As shown, the data processing device 100 may include at least a processing unit 110, a storage unit 120, an input / output (I / O) unit 130, and an image capturing unit 140. Note that in... Figure 1 The data processing device 100 shown is only for illustrating embodiments of the present invention, and the present invention is not intended to be construed as such. Figure 1 This is a limitation. The data processing device 100 may also include other components.

[0028] According to one embodiment of the present invention, the data processing device 100 may be a smartphone, a tablet computer, or a desktop computer, but the present invention is not limited thereto.

[0029] According to one embodiment of the present invention, the processing device 110 may be a general-purpose processor, a microcontroller unit (MCU), an application processor (AP), a digital signal processor (DSP), a graphics processing unit (GPU), a holographic processing unit (HPU), a neural processing unit (NPU), or other similar devices. The processing device 110 may include various circuits to provide the following functions: data processing and computation, storing data (e.g., program code) to and retrieving data from the storage device 120, transmitting a string of frame data (e.g., displaying text information, images, and videos, etc.) to a display device (not shown), and receiving user input and output signals via the input / output device 130. The processing device 110 can coordinate the above-described operations of the storage device 120, the input / output device 130, and the image capturing device 140 to perform the method of the present invention. Furthermore, according to an embodiment of the present invention, the processing device 110 can be used to run a Large Language Model (LLM) system.

[0030] According to one embodiment of the present invention, the storage device 120 may store software and firmware program code, system data, user data, etc. of the data processing device 100. The storage device 120 may be a volatile memory (e.g., random access memory, RAM), or a non-volatile memory (e.g., flash memory, read-only memory, ROM), a hard disk, or a combination of the above storage devices.

[0031] According to one embodiment of the present invention, the input / output (I / O) device 130 may include one or more buttons, a keyboard, a mouse, a touchpad, a microphone, and / or a speaker, etc., to serve as a human-machine interface (MMI) for user interaction. During LLM operation, the user can input queries and prompts into the LLM via the input / output device 130.

[0032] According to one embodiment of the present invention, the image capturing device 140 may be an electronic device with a shooting function, such as a camera. The image capturing device 140 can be used to generate image data that needs to be analyzed by LLM.

[0033] Figure 2 This is a schematic diagram illustrating the generation of image semantic data according to an embodiment of the present invention. Figure 2 The process for generating image semantic data shown is applicable to data processing device 100. For example... Figure 2 As shown, during Retrieval Augmented Generation (RAG), unstructured data and structured data (or vector data) of input data 210 can be obtained first. Input data 210 may come from an external database (e.g., an external artificial intelligence (AI) database, an external image database, an external audio-visual database, etc., but this invention is not limited thereto). Furthermore, input data 210 may contain multiple images, multiple videos, and multiple image tags, but this invention is not limited thereto. According to one embodiment of the present invention, the unstructured data corresponding to input data 210 may be image data, tag data, video data, or speech data. The structured data corresponding to input data 210 may be semantic data describing the input data.

[0034] In block 220, the unstructured and structured data corresponding to input data 210 can be integrated to produce integrated data (i.e., image semantic data). For example, if the input data is an automated optical inspection (AOI) image, the integrated data may include unstructured data (e.g., image data (e.g., an image of a printed circuit board, PCB) and markings on the image data (e.g., defect markings on the printed circuit board image), as well as structured data (e.g., semantic data describing relevant information about the printed circuit board in this image).

[0035] In block 230, after the integrated data is generated, it can be stored in a RAG structured database (e.g., a vector / structured query language (SQL) database, or a vector relational database) for use in subsequent LLM operations. Other vector / structured query language (SQL) data can also be stored in the RAG database.

[0036] Furthermore, according to one embodiment of the present invention, each piece of integrated data may correspond to a topic. Specifically, the integrated data corresponding to different topics may be grouped according to a clustering algorithm (e.g., a hierarchical clustering technique or a text clustering technique, but the present invention is not limited thereto). For example, a topic x may include the clustered image data and a dataset describing the image data. Topic x can be represented as x = {(I i D i |i=1...n)}, where I i D represents the image data of image i. i This represents the image data description corresponding to image i, where n is the size of the set. For example, in an AOI topic, image i could be a PCB image. A Large Language Model (LLM) can be used to list the main components of image data i, such as: "Green areas: Ground and power planes\n-Blue areas: Signal layers\n-White lines: Silkscreen\n-Yellow lines: Routingtraces\n-Orange lines: Keep-out areas\n-Black lines: Board outline". Therefore, the integrated data (image semantic data) recorded by image i can include image data I. i =PCB.jpg (i.e., image data of image i), and image description D i =["Green areas:Ground","Green areas:power planes","Blue areas:Signal layers","White lines:Silkscreen","Yellow lines:Routing traces","Orangelines:Keep-out areas","Black lines:Board outline"].

[0037] When grouping images, if the image descriptions of two images are too different, they will be grouped into different groups. Figure 3 This is a schematic diagram illustrating topic grouping according to an embodiment of the present invention. Figure 3 As shown, a PCB image 1 and its corresponding image description D 1,1 ~D 1,6 And an image of a coffee cup 2 and its description D 2,1 ~D 2,2Because the two images are too far apart (too similar), they will be grouped into different groups.

[0038] Figure 4 This is a word frequency diagram showing each topic according to an embodiment of the present invention. For example... Figure 4 As shown, to further optimize image description, a word frequency algorithm (e.g., Term Frequency-Inverted Document Frequency (TF-IDF) algorithm, but not limited thereto) can be used to calculate the score of the words contained in each topic, and the word with the highest score is selected as the best description. Figure 4 For example, in the topic "PCB", after word frequency analysis, "ground", "power planes", and "signal layers" are the three words with the highest scores (i.e., the highest frequency of occurrence). Therefore, these three words are selected as the best description of the topic "PCB".

[0039] Figure 5 This is a schematic diagram showing a program for a large language model (LLM) according to an embodiment of the present invention. Figure 5 The flowchart of the large language model (LLM) program shown is applicable to data processing device 100. For example... Figure 5 As shown, when performing RAG program 510, the unstructured data and structured data (or vector data) corresponding to the input data can be integrated into an integrated data (e.g., image semantic data) and stored in the RAG structured database for use in LLM program 520.

[0040] During an LLM procedure 520, a user can input a query (or a question and a prompt) into an LLM system. The LLM system can then convert the user-input query into vector data via an embedding operation. The LLM system can then retrieve relevant data matching the query from an LLM structured database (vector / structured query language database, or vector relational database) based on the vector data. The LLM structured database may contain a relational database management system (RDBMS).

[0041] Next, the LLM system can query the RAG structured database for integrated data of a subject that matches the vector data. The LLM system can then generate a retrieval result based on the data obtained from the LLM structured database and the integrated data obtained from the RAG structured database.

[0042] Next, the LLM system can sort the search results by similarity and generate a suggestion (i.e., an LLM suggestion) based on the similarity ranking. For example, the LLM system can use a top-K algorithm to sort the search results and use the most similar search result as a new LLM suggestion.

[0043] Next, the LLM system can predict the user's query based on the prompts from the new LLM system to generate a prediction result. The LLM system can use an agent to verify the accuracy of the prediction result. The agent can be an AI system designed for a specific topic, or another LLM system, but the present invention is not limited thereto.

[0044] If the prediction passes validation, the LLM system outputs a response based on the prediction result. If the prediction fails validation, the LLM system generates a new prompt.

[0045] The data processing apparatus 100 proposed in this invention can first generate integrated data (e.g., image semantic data) by integrating image and text information. When performing LLM operations, the LLM system can retrieve and classify data of different modalities. In other words, the LLM system will not need to process different types of data separately and then integrate the processed data of different types to obtain comprehensive information. Therefore, with the data processing apparatus 100 proposed in this invention, the LLM system can process image and text information more comprehensively, thereby improving the multimodal processing performance of LLM.

[0046] Furthermore, the data processing apparatus 100 proposed in this invention reduces the number of searches required by the LLM system in the RAG structured database. For example, given 100 images and 200 image descriptions, the integrated data generation method of this invention may produce integrated data corresponding to 10 topics. Therefore, when the LLM system searches the RAG structured database, it does not need to search all 100 images. The LLM system only needs to find integrated data that matches the query topic. Thus, the number of searches required by the LLM system in the RAG structured database is reduced, and the efficiency of the LLM system is improved. In addition, key descriptions can be recorded through word frequency and clustering techniques to reduce uninteresting regions and irrelevant descriptive words, thereby making the LLM system's response more accurate.

[0047] The data processing device 100 proposed in this invention can be applied not only to defect detection in automated optical inspection (AOI) but also to various other applications. For example, the data processing device 100 can combine X-ray images with a radiologist's textual description of the X-ray to generate integrated data, thereby improving the accuracy of diagnostic references. Furthermore, the data processing device 100 can also be applied in the field of autonomous vehicles. The data processing device 100 can integrate data and related information generated by vehicle-mounted visual sensors (e.g., cameras) and light- and laser-based sensors (LiDAR) to generate integrated data, thereby improving the accuracy of autonomous driving decisions.

[0048] Figure 6 This is a flowchart 600 showing a data processing method according to an embodiment of the present invention. Figure 6 The data processing method shown can be applied to the data processing device 100. For example... Figure 6 As shown, in step S610, a processing device 110 of the data processing device 100 can obtain a structured data and an unstructured data corresponding to each input data based on multiple input data.

[0049] In step S620, the processing device 110 can integrate structured data and unstructured data to generate integrated data corresponding to each input data.

[0050] In step S620, the processing device 110 may store the integrated data into a Retrieval Enhancement Generation (RAG) database to provide a Large Language Model (LLM) for use.

[0051] According to one embodiment of the present invention, in the data processing method, multiple input data include multiple images, multiple videos, and multiple image tags.

[0052] According to an embodiment of the present invention, in the data processing method, structured data may be semantic data corresponding to each input data, and integrated data may include input data and semantic data.

[0053] According to one embodiment of the present invention, in the data processing method, each piece of integrated data may correspond to a theme.

[0054] According to an embodiment of the present invention, in a data processing method, the processing device 110 can group integrated data corresponding to different topics according to a grouping algorithm.

[0055] According to one embodiment of the present invention, in the data processing method, the processing device 110 can obtain a query. Furthermore, the processing device 110 can generate vector data from the query via an embedding operation. Additionally, the processing device 110 can obtain a retrieval result corresponding to the vector data from data stored in a vector relational database and integrated data stored in a RAG database based on the vector data.

[0056] According to one embodiment of the present invention, in the data processing method, the processing device 110 can query the integrated data of a subject that corresponds to and matches the vector data in the RAG database. Furthermore, the processing device 110 can obtain search results based on the data stored in the vector association database and the integrated data of the corresponding topic.

[0057] According to an embodiment of the present invention, in the data processing method, the processing device 110 may perform a similarity ranking based on the search results. Furthermore, the processing device 110 may generate a prompt based on the similarity ranking result.

[0058] According to one embodiment of the present invention, in a data processing method, the processing device 110 may input queries and prompts into an LLM model to generate a prediction result.

[0059] According to one embodiment of the present invention, in the data processing method, the processing device 110 may use a proxy system to verify the prediction result. If the prediction result passes the verification, the processing device 110 may output a response based on the prediction result.

[0060] According to the data processing method proposed in this invention, structured and unstructured data can be integrated to generate integrated data (image semantic data) corresponding to different topics. During LLM operations, the LLM system only needs to retrieve integrated data that matches the topic requested by the user. Therefore, the number of searches performed by the LLM system can be reduced, and the retrieval efficiency of the LLM system can be improved.

[0061] The serial numbers in this specification and the claims, such as "first," "second," etc., are for convenience only and there is no sequential relationship between them.

[0062] The steps of the methods and algorithms disclosed in this specification can be directly applied to hardware and software modules or a combination of both by executing a processor. A software module (including execution instructions and related data) and other data can be stored in a data storage device, such as random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, optical disc read-only memory (CD-ROM), DVDs, or any other computer-readable storage media format in the art. A storage media can be coupled to a machine device, for example, a computer / processor (referred to as processor in this specification for convenience), through which the processor can read information (such as program code) and write information to the storage media. A storage media can integrate a processor. An application-specific integrated circuit (ASIC) includes the processor and the storage media. A user equipment includes an application-specific integrated circuit. In other words, the processor and storage media are included in the user equipment in a manner that is not directly connected to the user equipment. Furthermore, in some embodiments, any product suitable for computer programs includes a readable storage medium, wherein the readable storage medium includes program code associated with one or more of the disclosed embodiments. In some embodiments, the product of computer programs may include packaging materials.

[0063] The above paragraphs use multiple levels of description. Clearly, the teachings in this article can be implemented in many ways, and any particular architecture or function revealed in the examples is merely a representative case. Based on the teachings in this article, anyone skilled in this art should understand that the various levels revealed herein can be implemented independently or that two or more levels can be implemented in combination.

[0064] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the appended claims.

Claims

1. A data processing method, comprising: The processing unit of the data processing device obtains structured data and unstructured data corresponding to each of the above input data based on multiple input data. The aforementioned processing apparatus integrates the structured data and the unstructured data to generate integrated data corresponding to each of the aforementioned input data; and The aforementioned processing device stores the integrated data into a Retrieval Augmented Generation (RAG) database for use in Large Language Models (LLMs).

2. The data processing method as described in claim 1, wherein the plurality of input data includes a plurality of images, a plurality of videos, and a plurality of image tags.

3. The data processing method as described in claim 1, wherein the structured data is semantic data corresponding to each of the input data, and the integrated data includes the input data and the semantic data.

4. The data processing method as described in claim 1, wherein each of the above-described integrated data corresponds to a topic.

5. The data processing method as described in claim 4, wherein the data processing method further includes: Using the aforementioned processing device, the integrated data corresponding to different themes is grouped according to the grouping algorithm.

6. The data processing method as described in claim 1, further comprising: The inquiry was obtained through the aforementioned processing device; The above-mentioned processing device generates vector data from the above-mentioned query through an embedding operation; as well as Using the aforementioned processing device, based on the aforementioned vector data, retrieval results corresponding to the aforementioned vector data are obtained from the data stored in the vector-associative database and the integrated data stored in the RAG database.

7. The data processing method as described in claim 6, wherein generating the above-mentioned search results further includes: The aforementioned processing device queries the integrated data of the entity in the RAG database that corresponds to and matches the aforementioned vector data; and The above-mentioned search results are obtained by using the above-mentioned processing device based on the data stored in the above-mentioned vector association database and the above-mentioned integrated data corresponding to the above-mentioned topic.

8. The data processing method as described in claim 6, further comprising: The above processing device is used to sort the similarity results based on the above search results; as well as The aforementioned processing device generates a prompt based on the similarity ranking results.

9. The data processing method as described in claim 8, further comprising: The above-mentioned processing device inputs the above-mentioned query and prompts into the above-mentioned LLM model to generate prediction results.

10. The data processing method as described in claim 9, further comprising: The above-mentioned prediction results are verified using the agent system through the above-mentioned processing device; as well as If the above prediction results are verified, the above processing device will output a response based on the above prediction results.

11. A data processing apparatus, comprising: The processing device acquires structured and unstructured data corresponding to each of the multiple input data, and integrates the structured and unstructured data to generate integrated data corresponding to each of the input data; and Retrieve the Retrieval Augmented Generation (RAG) database and store the above integrated data for use in the Large Language Model (LLM).

12. The data processing apparatus of claim 11, wherein the plurality of input data includes a plurality of images, a plurality of videos, and a plurality of image tags.

13. The data processing apparatus of claim 11, wherein the structured data is semantic data corresponding to each of the input data, and the integrated data includes the input data and the semantic data.

14. The data processing apparatus of claim 11, wherein each of the integrated data corresponds to a topic.

15. The data processing apparatus of claim 14, wherein the processing apparatus groups the integrated data corresponding to different topics according to a clustering algorithm.

16. The data processing apparatus of claim 11, wherein the processing apparatus obtains a query, generates vector data by embedding the query, and obtains a retrieval result corresponding to the vector data from data stored in a vector-associative database and the integrated data stored in a RAG database based on the vector data.

17. The data processing apparatus of claim 16, wherein the processing apparatus queries the integrated data of the subject corresponding to the vector data in the RAG database, and obtains the search result based on the data stored in the vector association database and the integrated data corresponding to the topic.

18. The data processing apparatus of claim 16, wherein the processing apparatus performs similarity sorting based on the search results and generates a prompt based on the similarity sorting results.

19. The data processing apparatus of claim 18, wherein the processing apparatus inputs the query and the prompt to the LLM model to generate a prediction result.

20. The data processing apparatus of claim 19, wherein the processing apparatus uses a proxy system to verify the prediction result, wherein if the response passes verification, the processing apparatus outputs a response based on the prediction result.

Images