Multi-modal large model retrieval method and device for security check field

Abstract

The invention provides a security check field-oriented multi-modal large model retrieval method and device, electronic equipment, a storage medium and a program product. The method includes receiving a user input including a target text and a target image, and processing the user input in accordance with a retrieval enhancement process. Wherein a first image block to be recognized in a target image is retrieved in a security check database, a first retrieval result is obtained, and the first retrieval result comprises at least one first security check image and first text record data of an image block successfully matched with the first image block in the first security check image; then using a first cue word prompt large model to determine the cargo type in the first image block according to the first text record data in the first retrieval result, so as to obtain a first cargo identification result; and then prompting the large model by using a preset second cue word to generate first reply content for the target text in combination with the first cargo identification result.

Description

Technical Field

[0001] The present disclosure relates to the field of security inspection, and in particular to a multimodal large model retrieval method, device, electronic device, storage medium and program product for the field of security inspection. Background Art

[0002] With the increasingly severe global security situation and the continuous increase in security inspection needs, security inspection work is facing unprecedented challenges. The amount of data in security inspection scenarios has increased dramatically, and the data types have become increasingly diversified, including multiple modalities such as item scan images, text reports, video surveillance, and biometrics. These massive, heterogeneous, and multimodal data contain rich security clues and are valuable resources for data mining and intelligent analysis. At the same time, in recent years, multimodal and large model technologies have developed rapidly, opening up new ways for the comprehensive processing of massive structured and unstructured data. Multimodal technology can effectively improve the accuracy of joint retrieval of multimodal data with its good feature fusion and alignment capabilities of different modalities. Large models, with their excellent semantic understanding capabilities, show great potential in accurately capturing and parsing users' diverse and complex retrieval needs.

[0003] However, current technology still faces many challenges in optimizing the specific application scenario of security inspection. For example, in the field of security inspection, there will be many perspective images, and the data record content items of each perspective image are often many and complicated. Security inspection often involves various declaration forms. For another example, the requirements may be different in different security inspection scenarios. In some scenarios, it may only be necessary to determine whether there are various types of contraband, and in some scenarios (such as customs), it is necessary to further determine whether it is consistent with the declared goods, etc., and the types of contraband may be different under different security inspection requirements. Due to such complexity and particularity, there is currently a lack of a solution that can seamlessly connect to the security inspection field and can effectively integrate various heterogeneous security inspection data, so that the accumulated security inspection data can be fully utilized in the security inspection field to provide a convenient solution for the security inspection process. Summary of the invention

[0004] In view of this, the present disclosure provides a multimodal large model retrieval method, device, electronic device, medium and program product for the security inspection field. With the help of the large model, the diverse and personalized needs of users in the security inspection field can be captured, and with the help of multimodal technology, various heterogeneous security inspection data can be retrieved according to needs, making full use of the large amount of multimodal and heterogeneous data measured in security inspections, and providing powerful security clues for the security inspection process.

[0005] In a first aspect of the disclosed embodiment, a multimodal large model retrieval method for the security inspection field is provided, wherein the method comprises: receiving user input, the user input comprises target text and target image; and processing the user input according to a retrieval enhancement process.

[0006] The processing of the user input according to the retrieval enhancement process includes: extracting an image block to be identified from the target image to obtain at least one first image block; retrieving the first image block in a security inspection database to obtain a first retrieval result, wherein the first retrieval result includes at least one first security inspection image and first text record data of an image block in the first security inspection image that successfully matches the first image block, wherein the security inspection database is a database formed based on historical security inspection data, and the historical security inspection data at least includes security inspection images and text record data of security inspection images; inputting the first text record data in the first retrieval result to the large model, and using a preset first prompt word to prompt the large model to determine the type of goods in the first image block according to the input first text information to obtain a first goods recognition result; obtaining an overall goods recognition result based on the first goods recognition result corresponding to at least one first image block; inputting the overall goods recognition result and the target text to the large model, and using a preset second prompt word to prompt the large model to generate a first reply content for the target text according to the overall goods recognition result, and output the first reply content.

[0007] According to an embodiment of the present disclosure, the processing of the user input according to the retrieval enhancement process further includes: using at least one target detection expert model to identify the first image block to obtain a second cargo identification result, wherein the target detection expert model is a machine learning model based on image recognition. Accordingly, the obtaining of the overall cargo identification result based on the first cargo identification result corresponding to at least one of the first image blocks further includes: obtaining the overall cargo identification result based on the first cargo identification result and the second cargo identification result corresponding to at least one of the first image blocks.

[0008] According to an embodiment of the present disclosure, the at least one target detection expert model includes at least one contraband detection model, wherein each contraband detection model is used to detect a type of contraband.

[0009] According to an embodiment of the present disclosure, extracting the image blocks to be identified from the target image to obtain at least one first image block includes: determining the area to be identified in the target image based on the content of the target text, the area to be identified including the entire target image or a user-specified area in the target image; and segmenting at least one first image block from the area to be identified based on image texture features.

[0010] According to an embodiment of the present disclosure, the method further includes: using an intent recognition model to recognize the user intent of the target text; and determining the user intent category based on the user intent and a preset intent classification. Wherein, when the user intent category is a search enhancement related intent category, the user input is processed according to a search enhancement process, wherein the intent in the search enhancement related intent category includes recognizing goods in an image.

[0011] According to an embodiment of the present disclosure, the method further includes: when the user intention category is a general search intention category, processing the user input according to a general search process, wherein the intention in the general search intention category includes retrieving data from the security inspection database. Wherein, processing the user input according to the general search process includes: extracting the image block to be retrieved from the target image to obtain at least one second image block; retrieving the second image block in the security inspection database to obtain a second search result, wherein the second search result includes at least one second security inspection image, and the marking information and second text record data of the image block in the second security inspection image that successfully matches the second image block; using a preset third prompt word to prompt the large model to organize the second search result according to the target text and output it.

[0012] According to an embodiment of the present disclosure, searching the second image block in the security inspection database to obtain the second search result also includes: preprocessing the target text to determine search range information; searching the second image block in the security inspection database according to the search range information to obtain the second search result.

[0013] According to an embodiment of the present disclosure, the security inspection database includes a structured database and an unstructured database, wherein the unstructured database includes at least one of the following: a vector database or a graph database.

[0014] According to an embodiment of the present disclosure, the target image includes a perspective image, and the security inspection image includes a perspective image.

[0015] The second aspect of the disclosed embodiment provides a multimodal large model retrieval device for the security inspection field, which includes: a picture-text interaction module, a central scheduling module, an image segmentation expert module, a retriever, and a retrieval post-processing module.

[0016] The image-text interaction module is used to receive user input, where the user input includes target text and target image.

[0017] The central scheduling module is used to process the user input according to the retrieval enhancement process, wherein the central scheduling module calls the image segmentation expert module, the retriever and the retrieval post-processing module to process the user input.

[0018] Specifically, the image segmentation expert module is used to extract the image block to be identified from the target image to obtain at least one first image block.

[0019] The retriever is used to retrieve the first image block in the security inspection database to obtain a first retrieval result, wherein the first retrieval result includes at least one first security inspection image and first text record data of an image block in the first security inspection image that successfully matches the first image block; wherein the security inspection database is a database formed based on historical security inspection data, and the historical security inspection data at least includes security inspection images and text record data of security inspection images.

[0020] The post-retrieval processing module is used to: input the first text record data in the first search result into the large model, and use a preset first prompt word to prompt the large model to determine the type of goods in the first image block according to the input first text information to obtain a first goods recognition result, and obtain an overall goods recognition result based on the first goods recognition result corresponding to at least one of the first image blocks; and input the overall goods recognition result and the target text into the large model, and use a preset second prompt word to prompt the large model to generate a first reply content for the target text according to the overall goods recognition result.

[0021] Among them, the graphic and text interaction module is also used to output the first reply content.

[0022] According to an embodiment of the present disclosure, the device further includes various expert modules, wherein at least one target detection expert model is deployed in the various expert modules, and the target detection expert model is a machine learning model based on image recognition. The central scheduling module is also used to call the various expert modules. The various expert modules are used to use at least one target detection expert model to identify the first image block to obtain a second cargo identification result. The post-retrieval processing module is also used to obtain the overall cargo identification result based on the first cargo identification result and the second cargo identification result corresponding to at least one of the first image blocks.

[0023] According to one embodiment of the present disclosure, the central scheduling module is also used to: use an intent recognition model to identify the user intent of the target text; determine the user intent category based on the user intent and a preset intent classification, and when the user intent category is a retrieval enhancement related intent category, process the user input according to the retrieval enhancement process, wherein the intent in the retrieval enhancement related intent category includes identifying goods in an image.

[0024] According to an embodiment of the present disclosure, the central scheduling module is further used to: when the user intention category is a general search intention category, process the user input according to a general search process, wherein the intention in the general search intention category includes retrieving data from the security inspection database. The central scheduling module calls the image segmentation expert module, the retriever and the post-retrieval processing module according to the general search process to process the user input.

[0025] Specifically, the image segmentation expert module is used to extract the image block to be retrieved from the target image to obtain at least one second image block.

[0026] The retriever is used to retrieve the second image block in the security inspection database to obtain a second retrieval result, wherein the second retrieval result includes at least one second security inspection image, and marking information and second text record data of an image block in the second security inspection image that successfully matches the second image block.

[0027] The post-retrieval processing module is used to use a preset third prompt word to prompt the large model to merge and output the second search results according to the target text.

[0028] According to an embodiment of the present disclosure, the central scheduling module processes the user input according to a general search process and also calls a pre-search processing module. The pre-search processing module is used to pre-process the target text and determine the search range information. Accordingly, the retriever is specifically used to retrieve the second image block in the security inspection database according to the search range information to obtain the second search result.

[0029] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided. The electronic device includes: one or more processors, and a memory. The memory is used to store one or more computer programs. The one or more processors execute the one or more computer programs to implement the steps of the above method.

[0030] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program or instructions are stored, and when the computer program or instructions are executed by a processor, the steps of the above method are implemented.

[0031] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the steps of the above method are implemented.

[0032] One or more of the above embodiments have the following advantages or beneficial effects: the big model is used twice, the first time is to use the big model to organize the first search results, and the second time is to answer user questions through the big model. In this way, on the one hand, the initial results retrieved from the security inspection database are organized by the big model, so that the use of historical security inspection data to provide available and reliable security clues can be realized. On the other hand, the big model can be used to answer user questions based on the search results to feedback the results to the user, which can meet the diverse user needs in security inspection and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0034] Figure 1 The application scenario of the multimodal large model retrieval method and device for the security inspection field according to the embodiment of the present disclosure is schematically shown;

[0035] Figure 2 A flowchart of a multimodal large model retrieval method for the security inspection field according to an embodiment of the present disclosure is schematically shown;

[0036] Figure 3 A flowchart of a multimodal large model retrieval method for the security inspection field in another embodiment of the present disclosure is schematically shown;

[0037] Figure 4 A flowchart of a multimodal large model retrieval method for the security inspection field according to another embodiment of the present disclosure is schematically shown;

[0038] Figure 5 A schematic diagram schematically shows a general search process in an embodiment of the present disclosure;

[0039] Figure 6 A block diagram of a multi-modal large model retrieval device for the security inspection field according to an embodiment of the present disclosure is schematically shown;

[0040] Figure 7 A block diagram schematically shows a multi-modal large model retrieval device for the security inspection field according to another embodiment of the present disclosure;

[0041] Fig. 8A and Figure 8B A schematic diagram schematically shows a graphic-text interaction interface in an embodiment of the present disclosure;

[0042] Fig. 9 An example of feature extraction in the field of customs inspection is schematically shown;

[0043] Fig.10The target image input by the user in an example of a multimodal large model retrieval device for the security inspection field according to an embodiment of the present disclosure is schematically shown, wherein the retrieval enhancement processing flow is applied in the example;

[0044] Fig.11A and Fig. 11B Schematically shows Fig.10 An example of two groups of first search results obtained by the retriever in the example shown;

[0045] Fig.12 The target image input by the user in another example of the multimodal large model retrieval device for the security inspection field using the embodiment of the present disclosure is schematically shown, wherein the general retrieval process is applied in the example;

[0046] Fig.13 Schematically shows Fig.12 An example of a second search result obtained by the searcher in the example shown;

[0047] Fig.14 Schematically shows Fig.12 An example of the final result output to the user in the example shown;

[0048] Fig.15 A schematic diagram of an electronic device suitable for implementing the multimodal large model retrieval method for the security inspection field according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0049] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0050] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0051] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0052] Figure 1 The application scenario 100 of the multimodal large model retrieval method and device for the security inspection field according to an embodiment of the present disclosure is schematically shown.

[0053] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, a server 105 and a security inspection database 106. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0054] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as browsers, large model chat client applications, social software, etc. Among them, the multimodal large model retrieval device for the security inspection field of the embodiment of the present disclosure can provide a graphic interaction interface. The graphic interaction interface can be presented to the user through a browser or an application client in the terminal devices 101, 102, 103.

[0055] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0056] The server 105 may be a server that provides various services. A large model is set in the server 105.

[0057] The server 105 can communicate with the security inspection database 106. The server 105 can be provided with a multimodal large model retrieval device for the security inspection field according to an embodiment of the disclosure, through which the server 105 can analyze and process the input information sent by the user to the server 105 using the terminal devices 101, 102, and 103 according to the multimodal large model retrieval method for the security inspection field according to an embodiment of the disclosure, and feed back the processing result to the terminal device. The multimodal large model retrieval device for the security inspection field according to an embodiment of the disclosure is constructed based on multimodal and large model technologies, and can efficiently and accurately retrieve and deeply utilize massive security inspection multimodal heterogeneous data.

[0058] The security inspection database 106 can retrieve historical security inspection data from the security inspection database 106 according to the processing requirements during the process of processing user input on the server 105 to assist the big model in generating a better response. The data in the security inspection database 106 can effectively augment the knowledge of the big model. In particular, the search results are intelligently organized and optimized in combination with the generation and understanding capabilities of the big model, and the multimodal big model retrieval technology is used to provide knowledge augmentation for the big model, make up for its knowledge blind spots, and improve its overall ability to answer user input.

[0059] It should be noted that Figure 1 The application scenarios are merely exemplary and do not constitute any limitation on the application scenarios, architectures, and environments of the embodiments of the present disclosure.

[0060] Figure 2 The flowchart of a multimodal large model retrieval method for the security inspection field according to an embodiment of the present disclosure is schematically shown.

[0061] like Figure 2 As shown, the method may include operation S210 and operation S220.

[0062] First, in operation S210, user input is received, and the user input includes target text and target image. For example, through a picture-text interactive interface, the user is allowed to upload a picture and input the requirements through text. The target image can be a visible light image, a perspective image scanned during the security inspection process, or an image in video data. The target text can be data in text format or voice data that can be converted into text information.

[0063] Next, in operation S220, the user input is processed according to the search enhancement process.

[0064] Operation S220 may specifically include operations S221 to S226.

[0065] In operation S221, an image block to be identified is extracted from the target image to obtain at least one first image block.

[0066] In one embodiment, the target image may be determined to have a region to be identified based on the content of the target text, and the region to be identified includes the entire target image or a user-specified region in the target image. The target text is identified to determine whether the specified region is involved, and if so, an image block is extracted from the specified region; if not, an image block is extracted from the entire image.

[0067] When extracting the image block, at least one first image block is segmented from the area to be identified based on the image texture feature.

[0068] In operation S222, the first image block is retrieved in the security inspection database to obtain a first search result, wherein the first search result includes at least one first security inspection image and first text record data of an image block in the first security inspection image that successfully matches the first image block; wherein the security inspection database is a database formed based on historical security inspection data, and the historical security inspection data at least includes security inspection images and text record data of security inspection images. The text record data in the security inspection database may include cargo declaration information corresponding to the security inspection image and / or security inspection process record information of the cargo in the security inspection image, etc. The first text record data in the first search result comes from the text record data of the first security inspection image, and may be the corresponding item content (such as extracting cargo type or product name) extracted from the text record data of the first security inspection image according to the content of interest to the user in the target text.

[0069] In operation S223, the first text record data in the first search result is input to the large model, and a preset first prompt word is used to prompt the large model to determine the type of goods in the first image block according to the input first text information, so as to obtain a first goods recognition result. The large model can be arbitrary.

[0070] The reason why the large model is used to sort the first search results is that there may be many security inspection images for the same first image block in the first search results, and the goods in the image blocks that match the first image block in these security inspection images may not be completely consistent (for example, some are films, some are plastics, etc.). In this case, the large model is used to sort and count the goods information matched by the same first image block according to the first prompt word, and the goods with high credibility (such as selecting the one or several with the highest frequency) are selected as the goods recognition result of the first image block, thereby improving the credibility and readability of the first goods recognition result.

[0071] In operation S224, based on the first cargo identification result corresponding to at least one of the first image blocks, an overall cargo identification result is obtained. For example, the first cargo identification results corresponding to all the first image blocks are summarized. In other embodiments, the same type of cargo may be merged, sorted, etc. to form an overall cargo identification result.

[0072] In operation S225, the overall cargo identification result and the target text are input to the large model, and the preset second prompt word is used to prompt the large model to generate a first reply content for the target text according to the overall cargo identification result. The large model can combine the overall cargo identification result and the user question in the first text to answer the user's question, such as whether there are prohibited items or whether they are consistent with the declared goods.

[0073] In operation S226, the first reply content is output.

[0074] The method of the disclosed embodiment uses a large model twice. The first time is in operation S223, where the large model is used to sort the first search results and select goods with high credibility (such as selecting one or more with the highest frequency) as the goods identification result of the first image block. The second time is in operation S225, where the large model is used to answer user questions. In this way, on the one hand, the initial results retrieved from the security inspection database are sorted by the large model, making it possible to use historical security inspection data to provide available and reliable security clues. On the other hand, the large model can provide feedback to the user in the form of answering user questions based on the search results, which can meet the diverse user needs in security inspection and improve user experience.

[0075] The method of the embodiment of the present disclosure searches for security inspection clues from the historical security inspection data in the security inspection database, and can fully refer to the historical data to more comprehensively determine the types of goods in the image or image block to be identified. Compared with the solution directly through image recognition, the types and range of goods that can be identified in the embodiment of the present disclosure are wider. This is because the model for image recognition often depends on the types of items selected during training and annotation. If the model encounters a type of goods that has not been learned, it cannot be identified, resulting in a limited recognition range.

[0076] The method of the disclosed embodiment searches for similar items from the historical security inspection data in the security inspection database. When the amount of data in the security inspection database is sufficient, the comprehensiveness of the types of goods identification can be greatly improved. Specifically, with the increasingly severe global security situation and the continuous improvement of security inspection needs, the amount of data in security inspection scenarios has increased dramatically, and the data types have become increasingly diversified, including multiple modalities such as scanned images of items, text reports, video surveillance, and biometrics. These massive, heterogeneous, and multimodal data contain rich security clues and are valuable resources for data mining and intelligent analysis.

[0077] Figure 3 The flowchart of the multimodal large model retrieval method for the security inspection field in another embodiment of the present disclosure is schematically shown.

[0078] like Figure 3 As shown, the method may include operation S210 and operation S220, wherein operation S220 may include operation S2221 in addition to operations S221 to S226, and operation S224 is specifically operation S2241.

[0079] Operation S2221 may be performed in parallel with operation S222 and operation S223. In operation S2221, at least one target detection expert model may be used to identify the first image block to obtain a second cargo identification result, wherein the target detection expert model is a machine learning model based on image recognition. The target detection expert model identifies the type of cargo in the image by image recognition. The at least one target detection expert model may include, but is not limited to, detection models for various contraband (such as cigarettes, controlled knives, and liquid fuels).

[0080] Accordingly, in operation S2241, the overall cargo identification result is obtained based on the first cargo identification result and the second cargo identification result corresponding to at least one of the first image blocks. In some embodiments, the first cargo identification result and the second cargo identification result of the same first image block may be merged, wherein the inconsistent cargo types (such as product names) between the two are retained, and the frequency of occurrence of each product name may be recorded to provide a reference for subsequent models when generating reply content. In other embodiments, non-conflicting information in the first cargo identification result and the second cargo identification result of the same first image block may be retained, and in the case of conflict, the information in the second cargo identification result shall prevail. For example, if the expert model gives that a certain image block is a cigarette, and the retrieval result gives that it is a chair, in this case, the recognition result of the target detection expert model may prevail.

[0081] In this way, the first cargo identification result obtained in operation S223 and the second cargo identification result obtained in operation S2221 can be combined.

[0082] Next, in operation S225, the second prompt word is used to prompt the large model to generate a first reply content for the target text according to the overall cargo recognition result, and the first reply content is output in operation S226.

[0083] In this way, the disclosed embodiment can combine the retrieval of security clues from the security inspection database with the image recognition machine learning algorithm, and make up for the defect of limited recognition range in image recognition by retrieving from the security inspection database, thereby improving the comprehensiveness and accuracy of security inspection cargo identification.

[0084] It can be seen that the method of the disclosed embodiment can provide a new means for efficient, accurate retrieval and in-depth utilization of massive security inspection multimodal heterogeneous data based on cutting-edge multimodal large model retrieval technology, effectively innovating the traditional retrieval method in the field of security inspection. Among them, the traditional retrieval method generally relies on preset, static retrieval rules, mainly focusing on the simple matching of structured data, and lacks effective association retrieval and analysis methods for complex unstructured image data. More importantly, when security inspectors put forward complex and changeable query requirements, especially those queries that are not accurately matched with structured databases, such as queries based on fuzzy descriptions, contextual understanding or cross-modal associations, traditional retrieval methods often find it difficult to accurately capture the user's true intentions, lack flexibility and intelligence, and cannot effectively handle the diversity and uncertainty of user input, resulting in retrieval results that are often unsatisfactory, or even completely unable to meet the actual needs of users, and a large number of valuable unstructured data resources are therefore shelved and cannot be fully mined and utilized.

[0085] Figure 4 The flowchart of a multimodal large model retrieval method for the security inspection field according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 4 As shown, the method may include operation S210, operation S420 to operation S440, operation S220 and operation S450.

[0087] First, in operation S210 , a user input is received, where the user input includes a target text and a target image.

[0088] Next, in operation S420, the user intention of the target text is identified using an intention recognition model.

[0089] In operation S430, the user intent category is determined based on the user intent and the preset intent classification. For example, the user intent is divided into multiple types, including but not limited to the following main categories: general retrieval intent, retrieval enhancement related intent and general intent. The actual classification can be further refined according to system requirements. Among them, the general retrieval intent includes the user's intention to hope that the big model will retrieve data from the security inspection database on their behalf. The retrieval enhancement related intent includes the intention to identify the goods in the image. General intent is, for example, a conventional question-and-answer intent, such as a situation where the big model can answer without the help of the security inspection database.

[0090] Next, the user input is processed using a corresponding processing flow in a targeted manner according to the user intention category determined in operation S430 .

[0091] Specifically, when the user intention category is a search enhancement related intention, operation S220 is performed to process the user input according to the search enhancement process. The specific implementation of operation S220 can be referred to the above description, which will not be repeated here.

[0092] When the user intent category is a general search intent, operation S440 is performed to process the user input according to the general search process. The processing process of the general search process can be referred to below. Figure 5 Detailed introduction in.

[0093] When the user intent category is other intent categories (such as general intent) other than retrieval enhancement related intent and general retrieval intent, operation S450 is performed to process the user input according to other processing flows, such as processing the user input according to a question-and-answer dialogue flow.

[0094] Figure 5 The following is a schematic diagram of a general search process in an embodiment of the present disclosure.

[0095] like Figure 5 As shown, the specific process of processing the user input according to the general search process in operation S440 may include operations S441 to S443.

[0096] First, in operation S441, an image block to be retrieved is extracted from the target image to obtain at least one second image block.

[0097] Then, in operation S442, the second image block is retrieved in the security inspection database to obtain a second search result, where the second search result includes at least one second security inspection image, and marking information and second text record data of an image block in the second security inspection image that successfully matches the second image block;

[0098] Then, in operation S443, the preset third prompt word is used to prompt the large model to sort and output the second search result according to the target text. For example, according to the items required to be output in the target text, the items in the second text record data are screened, and then the retrieved second security inspection image containing the marking information and the screened text record data are classified and sorted and then output.

[0099] It can be seen that the method of the embodiment of the present disclosure can selectively process the process of processing user input according to the user's intention, and meet the diverse needs of users through a set of systems.

[0100] Figure 6 A block diagram of a multimodal large model retrieval device 600 for the security inspection field according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 6As shown, the device 600 may include a graphic-text interaction module 610 , a central scheduling module 620 , an image segmentation expert module 630 , a retriever 640 and a post-retrieval processing module 650 .

[0102] The image-text interaction module 610 is used to receive user input, where the user input includes a target text and a target image. In one embodiment, the image-text interaction module 610 may perform the operation S210 described above.

[0103] The central scheduling module 620 is used to process the user input according to the retrieval enhancement process, wherein the central scheduling module 620 calls the image segmentation expert module 630, the retriever 640 and the post-retrieval processing module 650 to process the user input.

[0104] Specifically, the image segmentation expert module 630 is used to extract the image block to be identified from the target image to obtain at least one first image block. In one embodiment, the image segmentation expert module 630 may perform the operation S221 described above.

[0105] The retriever 640 is used to retrieve the first image block in the security inspection database to obtain a first search result, wherein the first search result includes at least one first security inspection image and first text record data of an image block in the first security inspection image that successfully matches the first image block; wherein the security inspection database is a database formed based on historical security inspection data, and the historical security inspection data at least includes security inspection images and text record data of security inspection images. In one embodiment, the retriever 640 may perform the operation S222 described above.

[0106] The post-retrieval processing module 650 is used to: input the first text record data in the first retrieval result to the large model, and use the preset first prompt word to prompt the large model to determine the type of goods in the first image block according to the input first text information to obtain a first goods recognition result, and obtain an overall goods recognition result based on the first goods recognition result corresponding to at least one of the first image blocks; and input the overall goods recognition result and the target text to the large model, and use the preset second prompt word to prompt the large model to generate a first reply content for the target text according to the overall goods recognition result. In one embodiment, the post-retrieval processing module 650 can perform the operations S223, S224, and S225 described above.

[0107] The text-image interaction module 610 is further configured to output the first reply content. In one embodiment, the text-image interaction module 610 may perform the operation S226 described above.

[0108] The device 600 can realize reference Figure 2~Figure 4The described method can be specifically referred to in the previous article and will not be described in detail here.

[0109] The multimodal large model retrieval method and device of the disclosed embodiment innovates the traditional retrieval method, intelligently analyzes the diverse needs of users through an interactive operation interface, and plans the optimal retrieval path accordingly. Specifically, its advantages include:

[0110] - Efficient multimodal data processing and fusion: Utilize advanced multimodal data processing technology to achieve deep fusion of different modal information such as text, images, and videos in security inspection data, and provide comprehensive security clue analysis.

[0111] - Accurate and intelligent retrieval: Intelligently analyze user needs and provide accurate retrieval results, ensuring that security personnel can quickly retrieve the required information from massive data and improve work efficiency.

[0112] -Large model knowledge augmentation: Combined with the generation and understanding capabilities of large models, the search results are intelligently organized and optimized, and multimodal large model retrieval technology is used to provide knowledge augmentation for the large model, fill in its knowledge blind spots, and improve its overall answering ability.

[0113] -User-friendly interactive design: An intuitive and convenient interactive interface is designed to enable security personnel to easily input query requests, view search results, and enjoy a smooth operating experience.

[0114] Figure 7 The structural block diagram of a multimodal large model retrieval device 700 for the security inspection field according to an embodiment of the present disclosure is schematically shown.

[0115] like Figure 7 As shown, the device 700 may include a graphic-text interaction module 610, a central scheduling module 620, an image segmentation expert module 630, a pre-retrieval processing module 760, a retriever 640, a post-retrieval processing module 650, various expert modules 770, a generator 780 and a data management module 790.

[0116] Among them, the device 700 integrates multimodal large model retrieval technology and large model technology, and introduces the retrieval enhancement (RAG) mechanism, which can deeply mine and efficiently integrate the complex multimodal information in various security inspection scenarios, including but not limited to various scanned images of container vehicles, luggage and parcels, radioactive detection information, declaration information, visible light images, video images, voice, various related databases, etc. Through cross-modal efficient retrieval capabilities, not only the rapid positioning and extraction of these information resources are achieved, but also these valuable data can be used as core references and seamlessly connected to other non-retrieval function modules, significantly improving the accuracy and efficiency of these modules in processing complex tasks. The specific description of each module of the device 700 is as follows.

[0117] The data management module 790 is responsible for providing comprehensive, accurate and diverse data sources to support the various functions of the retrieval system. These data sources include, but are not limited to, security inspection cargo information, regulations, historical inspection records, filing information, container vehicles, luggage scan images, radioactive material detection results, and other types of structured, semi-structured and unstructured data. In order to reduce the cost of repeated retrieval, the data management module can also include the management of historical retrieval records.

[0118] The main functions of the data management module 790 include data source integration, data storage, data preprocessing and security inspection database.

[0119] Data source integration: Extract data from multiple heterogeneous data sources, clean, transform and load them, and integrate them into a unified data management platform. This process ensures the integrity, consistency and availability of the data.

[0120] Data storage: A hybrid storage strategy is used based on data types and access requirements. Structured data is stored in relational databases to facilitate SQL queries; unstructured data (such as documents, images, and audio) is stored in distributed file systems or object storage to support efficient data access and expansion.

[0121] Data preprocessing: Preprocess the integrated data, including data cleaning (noise removal, error correction), data formatting (unified coding, unified format), etc., to improve data quality and retrieval efficiency.

[0122] Security inspection database: including vector database, structured database, graph database, etc.

[0123] The image and text interaction module 610 can receive user inputs such as images, texts, and voices through an interactive interface, and output retrieved images, texts, and voices. The interactive interface can be as follows: Fig. 8A or as shown in 8B, wherein Fig. 8AThe interface shown uses the input form of images and keywords. Figure 8B The interface shown adopts the form of dialogue input.

[0124] Users can search for the entire image, a specific area in the image, input text to search, search by combining image input with questions, etc. The system supports outputting one or more images, outputting images and marking specific areas, various related information, plain text or voice, tables, etc.

[0125] The central scheduling module 620 is the core hub of the device 700 and can interact with other modules.

[0126] The central scheduling module 620 adopts natural language processing (NLP) technology, such as text classification, named entity recognition, etc., and can also directly use large models to achieve accurate analysis and flexible classification of user intentions, divide user intentions according to the user's specific needs and the complexity of the needs, and customize differentiated processing procedures and routing strategies for each intent.

[0127] User intent classification: Based on a deep understanding of user input content, user intent is classified into multiple types, including but not limited to the following main categories: general search intent, search enhancement related intent and general intent. The actual classification can be further refined according to system requirements.

[0128] General search intent: For direct query requests from users in a variety of ways such as images and texts, the central scheduling module 620 quickly identifies and guides them to the pre-search processing module 760, and uses efficient search algorithms in the retriever 640 to accurately extract relevant information from the massive database to meet the user's immediate needs.

[0129] Retrieval enhancement related intent: covers a series of query scenarios that require complex processing and in-depth analysis. When the questions raised by users require high security inspection expertise or require more security inspection knowledge assistance, or when the existing expert modules (such as target detection models, image description models, risk analysis models, etc.) and generators (such as general language large models) cannot directly provide satisfactory answers to the questions raised by users, the central scheduling module 620 will classify such queries as retrieval enhancement related intent.

[0130] General intent: For user questions that can be answered directly and accurately without the help of data provided by the data management module, the central scheduling module 620 can classify them as general intent. The processing flow of this type of intent is simple, and internal resources (such as various expert modules 770, generators (including large models), etc.) can be directly called to answer.

[0131] The central scheduling module 620 routes the user input to different processing flows according to the user's intent. Specifically, after identifying the user's intent, the central scheduling module 620 can intelligently plan and execute the optimal processing path according to different intent categories.

[0132] For general search intentions: the central scheduling module 620 can use efficient retrieval algorithms to retrieve relevant information from the database or information source after pre-search processing of user questions (such as query optimization, keyword expansion, etc.). For complex queries that need to be executed step by step, the central scheduling module will plan detailed execution steps and ensure that each step can return valuable information to provide support for subsequent operations.

[0133] For search enhancement intention: the central scheduling module 620 starts the search enhancement process. The process includes retrieving relevant data from multiple information sources according to the planned path, integrating relevant information, calling expert modules to assist in judgment, performing multiple rounds of evaluation and optimization, etc., so as to provide more comprehensive, in-depth and accurate answers.

[0134] For general intents: the central scheduling module 620 will directly use resources other than the data management module 790 (such as various expert modules, large models in the generator, etc.) to answer, ensuring the simplicity and efficiency of the processing flow.

[0135] The retrieval pre-processing module 760 is an important component of the retrieval. It is responsible for preprocessing user queries or retrieval requests to ensure that the query statements can be processed accurately and efficiently by the retriever. It includes deep analysis of user queries, using synonym dictionaries, machine learning models, knowledge graphs, or large models and other technologies to enrich and expand query statements, optimize and reconstruct query statements based on enhanced results, and other operations.

[0136] One of the functions of the pre-search processing module 760 is query enhancement. Through strategies such as keyword expansion, synonym replacement, pronoun replacement, and concept association, the query content is enriched, the search scope is broadened, and the possibility of finding relevant information is increased, providing users with more comprehensive and rich search results. For example, if a user asks to find an image declared as fruit at the customs, first the customs is replaced with the customs area on the declaration document, and then the fruit is expanded to various specific fruit categories such as apples, pears, bananas, and pineapples.

[0137] The second function of the pre-retrieval processing module 760 is query construction: converting the enhanced query intent into a query statement that can be executed by the retriever. This module can parse query requirements from different modalities, including visible light images, perspective images, videos, voices, and texts. When extracting keywords or phrases, intelligent classification is performed according to the different retrieval methods of the retriever. For example, it is identified whether it is necessary to perform image vector retrieval and which image vector retrieval method to call, whether to return images as part of the result set, which text queries need to be split for vector retrieval, and which can be directly converted into SQL query statements to perform structured queries.

[0138] Retriever 640 is designed as a multi-source data comprehensive retrieval engine in the security inspection application scenario. It can integrate multiple database resources such as vector libraries and structured databases, and integrate multiple retrieval strategies, including but not limited to keyword exact matching, vector similarity calculation, and graph-based semantic association analysis, etc. It performs refined retrieval operations for user queries and supports multi-condition joint retrieval, and each retrieval item can be assigned different weights. In particular, for data types unique to security inspection (such as cargo lists, container scanned images, etc.), the system supports highly customized retrieval requirements, effectively improving retrieval efficiency and effectiveness.

[0139] The core modules of the retriever 640 include a feature extraction module and an indexing module.

[0140] The feature extraction module in the retriever 640 integrates a variety of feature extraction models to cope with the complex requirements of different information sources. Fig. 9 The following schematic diagram shows an example of feature extraction in the field of customs inspection. Fig. 9 The feature extraction module is described as follows.

[0141] Combination Fig. 9For text data, in order to cope with complex requirements in specific fields, such as professional terminology recognition and policy clause analysis in security inspection documents, a text feature extraction model customized for security inspection scenarios is used based on natural language processing technology to extract features of texts such as keywords, phrases, and sentences, ensuring that key information in the text can be accurately captured. For image and audio data, deep learning models are used to extract visual and audio features and convert them into vector form. In order to cope with complex images in specific fields, such as a large number of container vehicle scan images, luggage X-ray images, CT images, etc. in the customs inspection field, special feature extraction models such as cargo feature extraction models, vehicle head feature extraction models, and chassis feature extraction models are used according to business needs to extract corresponding image features. Through multimodal learning frameworks, such as cross-modal pre-training models, data from different modalities can also be processed simultaneously, so that these features are represented in a unified feature space, which is convenient for subsequent multimodal large model retrieval. For all data with the same serial number in the security inspection scenario, the relevant feature vectors are associated with the structured data through the serial number.

[0142] The index module in the retriever 640 converts the multimodal data in the data source into an efficient index structure to quickly respond to retrieval requests. For text data, the inverted index technology is used to associate keywords with document IDs; for data such as images and videos, the vector index technology is used to construct an approximate nearest neighbor search index in the vector space for each vector feature extracted by the feature extraction module. In combination with multimodal features, a hybrid index library can be constructed, and through a hybrid index strategy, efficient organization and retrieval of multimodal data in a unified framework can be achieved. Update the index regularly to reflect changes in the data source. Implement an incremental indexing mechanism to update the index only for newly added or modified data, reducing index construction time and resource consumption. Monitor index performance, optimize index structure and query algorithms, and ensure the accuracy and efficiency of the index.

[0143] The post-retrieval processing module 650 performs post-processing on the results returned by the retriever 640 to improve the quality of the retrieval results and the user experience, including sorting, deduplication, adding prompt words, etc. The post-retrieval processing module 650 can use machine learning algorithms, information retrieval evaluation indicators and user feedback data for specific security inspection tasks. In the case of unsatisfactory results of a single query, the system can perform multiple iterative retrievals and further fine-tune and optimize the retrieval results. The functions of this module include but are not limited to the following: result integration, sorting optimization and adding prompt words.

[0144] Results integration includes the cross-modal integration of search results from different modalities (such as text, image, audio, video, etc.) by the search post-processing module 650 to form a comprehensive, multi-dimensional result set to help users understand the query content from multiple perspectives and improve the comprehensiveness and accuracy of the search. It also includes the function of removing duplicate and irrelevant results to reduce noise interference and ensure that the returned result set is high-quality and valuable.

[0145] Sorting optimization refers to the post-retrieval processing module 650 sorting the search results according to the matching degree with the user's query intent through a refined correlation calculation mechanism. Specifically, for different types of query intent, such as "what is in the image" and other image description-related search enhancement questions, the image-text matching model is trained to calculate the correlation between the user-input image and the retrieved text, so as to re-rank the search results; for general search requirements such as "retrieve fruit-related documents", the semantic matching model is trained to calculate the correlation between the retrieved documents and the user's questions, etc. Through this differentiated correlation calculation method, the module can ensure that the most relevant and valuable results are always ranked at the top, significantly improving the user's search efficiency and satisfaction.

[0146] Adding prompt words: For retrieval enhancement intent, the post-retrieval processing module 650 will analyze the query content and add necessary prompt words or context information to enrich the semantic expression of the retrieval results and improve the input quality of the generator model.

[0147] Generator 780 generates answers or summaries that meet user needs based on the search results. A large model can be used to combine corresponding prompt words according to different user intentions to generate a final conclusion based on the search results.

[0148] Next, two embodiments are used to illustrate the specific use of the device 700 in the field of security inspection.

[0149] Fig.10 The target image input by the user in an example of the multimodal large model retrieval device for security inspection field using the embodiment of the present disclosure is schematically shown. The example is an example of retrieval enhancement requirement in a customs image review scenario, and the retrieval enhancement processing flow is applied.

[0150] In this example, users can input text and images through the graphic interface. The text input is: Does the cargo in the container match the customs declaration information? Customs declaration cargo: film, and the image input is: Fig.10 Image shown.

[0151] The search source is the security inspection database in the data management module 790. The vector database uses FAISS to manage the perspective image of the container and its regional features, and the text features of the relevant information corresponding to the perspective image. The structured database uses MySQL to manage the declaration information, and the feature extraction module is used to extract features from the perspective image and text information respectively. Each perspective image is bound to the declaration information in the structured database through the image number.

[0152] The intent recognition model in the central scheduling module 620 outputs the user intent as a retrieval enhancement related intent category, so that according to the input image, the process is determined as follows: the central scheduling module 620 calls the image segmentation expert module to divide the image into different texture areas, extracts the features of each texture area (i.e., image block), and calls the retriever 640 to retrieve similar images corresponding to each texture area, and then calls the retrieval post-processing module 650 to complete the category of the goods name of the current image based on the text description of the similar image. At the same time, the central scheduling module 620 also calls various expert modules 770 (such as the contraband detection expert module) to give reference conclusions, and inputs the retrieval enhancement conclusions and the conclusions of the contraband detection model into the generator 780, which generates the final conclusion for the user question. The following is an explanation of the retrieval enhancement process.

[0153] Retriever 640: Use the feature extraction module for container vehicle images to extract image features, including but not limited to local features extracted from different texture areas of the goods in the input image. Then use the extracted features to perform vector retrieval to obtain goods images similar to the local features (i.e., the first security inspection image), and return an image list and the declaration information corresponding to the image. Each image returns similar images of each texture area and corresponding text record data such as product name and tax number. Among them, Fig.11A and 11B For the retriever Fig.10 An example of search results returned in two different regions.

[0154] The post-retrieval processing module 650 then processes the search results provided by the retriever 640 . The search results after processing should also be combined with the recognition results of various expert modules 770 .

[0155] The post-search processing module 650 first processes the search results. Fig.11A and 11B The multiple groups of product names shown in the figure each group of product names corresponds to a texture area in the image. The post-retrieval processing module 650 first determines the product name of each texture area, and then outputs all product names of the entire image. Specifically, a specific algorithm or a generation model can be used to merge, remove duplicates, reorder, etc. all returned product names.

[0156] For example, the search results are merged using the generator 780 and the preset first prompt word. An example of the first prompt word is as follows:

[0157] "You are a customs declaration specialist. According to the classification method of the product name in the customs tariff, you classify and count the product names. The classification and statistics method is: first, merge the product names of the same type according to the customs tariff, then count the frequency of each product name, rank them according to the frequency of occurrence, and output the top k product names. Several groups of product names are given below. First, perform classification statistics on each group, k=1, and merge the classification statistics results of each group; then form a new group based on the classification statistics results, and then perform classification statistics on the newly formed group, k=5. At this time, if the returned product name categories are less than 5, all product name categories are output.

[0158] Group 1: List of goods names

[0159] Group 2: List of goods names

[0160] …

[0161] Group n: List of goods names

[0162] Output product name:

[0163] The first prompt word and Fig.11A and Fig. 11B After combining the two sets of search results, the list of goods names is instantiated as follows:

[0164] "Group 1: Film, film, plastic, film

[0165] Group 2: plastic moulds, plastic moulds, polyethylene moulds, chairs;”

[0166] The first prompt word and Fig.11A and Fig. 11B The result is provided to generator 780, which firstly performs statistics on the product names matched by each image block to select the product names with the highest frequency, and then combines the product names selected by all image blocks into a new group, from which the five product names with the highest frequency are selected as the product identification result and the output result is: film, plastic mold.

[0167] In this way, the large model in the generator 780 can be used to sort and count the matched cargo information according to the first prompt word, and the cargo names with high credibility can be screened out as the first cargo identification result.

[0168] Next, the post-retrieval processing module 650 may further combine the processed retrieval result (ie, the first cargo identification result) with the result of the contraband detection expert module, and then generate reply information to the user through the generator 780 .

[0169] For example, suppose the Contraband Detection Expert module is Fig.10 Then, the second prompt word is used to integrate the output of the contraband detection expert module and the retrieved goods name list retrieved by the retriever 640 and processed by the generator 780. An example of the second prompt word is as follows:

[0170] "The cargo in the container has a texture similar to the following cargo: Retrieve cargo name list

[0171] The container contains the following prohibited items: Result of the Prohibited Items Detection Expert Module

[0172] Customs declaration goods: List of goods for customs declaration

[0173] Based on the above information, determine what goods are in the container and answer the user's questions

[0174] User question: Do the goods in the container match the customs declaration information?

[0175] Output:"

[0176] In this example, the second prompt word is instantiated as:

[0177] "The cargo in the container has a texture similar to the following cargoes: film, plastic moulds

[0178] The container contained the following prohibited items: Cigarettes

[0179] Goods declared for customs: Film".

[0180] Finally, the generator outputs the reply content, which can be:

[0181] “The container does contain goods that match the customs declaration information (film), but there may also be goods that do not match (cigarettes, plastic molds).”

[0182] In this example, the generator is called twice, the first time to integrate the search results, and the second time to answer the user's question. According to the generator selected in this example, the output results of the two calls to the generator are:

[0183] First integrated search results: Output product name: film, plastic mold

[0184] Second answer to the user's question: The container does contain goods (film) that are consistent with the customs declaration information, but there may also be non-compliant goods (cigarettes, plastic molds).

[0185] It should be noted that in the search enhancement process, if Fig.11A and 11BIf the user does not explicitly request the return of the search result (such as in this example), the search result will not be provided to the user through the graphic and text interactive interface.

[0186] Fig.12 The target image input by the user in another example of the multimodal large model retrieval device for the security inspection field using the embodiment of the present disclosure is schematically shown, wherein the general retrieval process is applied in the example.

[0187] This example is an example of image-text retrieval in a customs image review scenario. The user input is received through the image-text interaction interface. The text input is "search for the boxed goods exported from this port last month", and the image input is Fig.12 Image shown.

[0188] The search source is the security inspection database in the data management module 790. The vector database uses FAISS to manage the perspective image of the container and its regional features, and the text features of the relevant information corresponding to the perspective image. The structured database uses MySQL to manage the declaration information, and the feature extraction module is used to extract features from the perspective image and text information respectively. Each perspective image is bound to the declaration information in the structured database through the image number.

[0189] The intent recognition function in the central scheduling module 620 outputs the intent as a general search intent, such as image and text search, so that a general search process is executed according to the input text, image and specified box.

[0190] The retrieval pre-processing module 610 can pre-process the input text and determine the retrieval scope information. Specifically, the query enhancement function can be used to replace the reference words in the query statement, such as replacing the current port with port A, and last month is June 2024. The Text2SQL model is used to convert the user input into a query vector library and a query SQL library. Identify the need for similar image retrieval of container cargo images in the query statement, and convert the port and time query information in the query statement into an SQL query statement. The converted SQL query statement is:

[0191] SELECT *

[0192] FROM history table

[0193] WHERE export port = 'Port A'

[0194] AND export time BETWEEN '2024-06-01' AND '2024-06-30'.

[0195] The retriever 640 uses the feature extraction module of the container vehicle image to extract the features in the input image frame, performs feature vector retrieval, obtains the cargo image similar to the local features, and returns the image list and the declaration information corresponding to the image. In the data returned by the vector library, the SQL statement generated by the retrieval pre-processing module 760 is used to further filter the data that meets the conditions. In the example, the filtered data only contains the cargo name, export time and port name. The actual data may include cargo HSCODE, cargo name, cargo specifications, country of origin, enterprise, import / export port and other related information bound to the image. Fig.13 Shown are the top 4 cargo images and corresponding text record data returned by the retriever 640 .

[0196] The post-retrieval processing module 650 interacts with the generator 780 to merge, reorder and perform other operations on the retrieval results to ensure that the returned results meet the user's query requirements.

[0197] In this example, the third prompt word is used to merge the search results. According to the classification requirements of the customs tariff, the names of the products returned by the search are classified, and the frequency of each category is counted. They are re-sorted according to the frequency. The top three names are selected and the image-text matching model is used to further determine the image-text relevance. Finally, the highest-ranked category is output as the recommended name of the goods searched by the user. The output content can include text and images, where the output text can be "The goods in the box may be film, and the list of similar images is as follows", and the output image is Fig.14 The output text and image are combined and returned to the user through the graphic-text interactive interface.

[0198] According to the embodiment of the present disclosure, the system process of the device 700 is divided into two core links: data storage and user retrieval. The two work closely together to support an efficient and intelligent information processing system.

[0199] Introduction to the data storage process

[0200] Data storage is the foundation for building an information retrieval system, which aims to integrate and convert diverse data sources (including security inspection images, document text, visible light images, videos and other unstructured data) into a searchable format. This process is divided into two modes: batch and single data storage to meet the needs of data processing of different scales.

[0201] Data collation: Clean, standardize and classify the collected data in the data management module to ensure data quality and consistency.

[0202] Feature extraction: Utilize various feature extraction modules in the retriever to extract key information from images, texts and other data to generate feature vectors or keywords for retrieval.

[0203] Data storage: The processed data and its characteristic information are stored in the corresponding database to support fast retrieval and efficient access. At the same time, an incremental update strategy is implemented to ensure the timeliness and accuracy of the database content.

[0204] User search process: The user search process is a direct reflection of the system's value. Through a series of refined operations, it converts user query needs into accurate responses or suggestions.

[0205] Introduction to user search process.

[0206] Graphics and text interaction: Users input query questions in the form of images or text through a friendly graphics and text interaction interface, and the system ultimately provides system answers in the form of images and text.

[0207] Central Scheduling: The central scheduling module intelligently analyzes the intent of user input and assigns the optimal execution path for different types of queries (such as retrieval questions, retrieval enhancement questions, etc.).

[0208] Pre-retrieval processing: For queries that require retrieval or retrieval enhancement, the pre-processing module expands, rewrites, completes, and other operations on the query request to generate a more accurate query statement that is more suitable for the retriever to process.

[0209] Retrieval execution: The retriever performs retrieval operations in multiple data sources based on the processed query statements. For vector retrieval requests, query features are specially extracted and efficient vector retrieval algorithms are used to quickly locate relevant information.

[0210] Retrieval post-processing and enhancement: The post-processing module integrates and sorts the retrieval results. For general retrieval intents, the output of this module can also be directly used as the user's final output. For retrieval enhancement intents, this module also needs to add necessary prompt words to provide high-quality input for the generator.

[0211] Generate responses: Based on a deep learning model, the generator combines prompt words, query results, and the user's original request to generate natural and fluent responses or suggestions that meet user needs.

[0212] In the above process, the system records user queries and their search results as feedback data for continuous optimization of various models in the search system. At the same time, for new data not included in the search source, the system will perform necessary data storage operations to enrich data resources and improve search results.

[0213] Through the above process, the device 700 not only realizes efficient and accurate information retrieval and response generation, but also continuously improves its own intelligence level and user experience through continuous feedback and update mechanism.

[0214] The multimodal large model retrieval method and device of the disclosed embodiment have brought significant technical progress and operational optimization in the field of security inspection. The specific beneficial effects can be summarized as follows:

[0215] 1. Optimize interactive experience and improve work efficiency

[0216] The disclosed embodiment innovatively introduces a text-and-picture dialogue interaction mode, so that security inspectors do not need to perform complex database query operations, and can only retrieve information and issue instructions through natural language dialogue. This intuitive and convenient interaction mode greatly reduces the operation threshold, significantly improves work efficiency, and also reduces the workload of security inspectors.

[0217] 2. Strengthen data processing capabilities and expand the scope of information utilization

[0218] The disclosed embodiments provide powerful data processing capabilities and can efficiently process massive amounts of structured and unstructured data. In particular, for unstructured data (such as images, videos, etc.) that were previously difficult to effectively utilize, the system has achieved efficient integration and utilization through multimodal large model retrieval technology, providing richer and more comprehensive information support for security inspections. Combined with the deep learning capabilities of large models, this system can further explore the associations and patterns behind the data and provide security inspectors with more in-depth and accurate retrieval results. This deep retrieval capability helps to discover potential risk points and improve the accuracy of inspections.

[0219] 3. Make up for the knowledge gap of large models and solve the problem of local knowledge base

[0220] For scenarios where a large model is used as a generator, the disclosed embodiment uses retrieval enhancement technology to effectively supplement the large model with massive security inspection data. In particular, for complex scenarios such as perspective images, the system can provide necessary background information and reference basis for the current image by retrieving and analyzing the content of historical images, thereby improving the intelligence level and practicality of the large model in security inspection.

[0221] 4. Improve comprehensive risk identification capabilities

[0222] The disclosed embodiment can quickly identify various associations and potential risks related to the current goods through rapid retrieval and retrieval enhancement technology, helping security inspectors to prevent, quickly discover and deal with various high-risk situations. At the same time, the system can also perform intelligent analysis based on real-time data and historical cases, providing more accurate and scientific decision-making support for security inspection work, thereby further improving security inspection efficiency and safety.

[0223] To sum up, the multimodal large model retrieval method and device of the disclosed embodiments have shown significant beneficial effects in the field of security inspection. It not only optimizes the interactive experience and improves work efficiency, but also strengthens data processing capabilities and broadens the scope of information utilization. At the same time, it makes up for the knowledge deficiencies of large models, enhances the level of intelligence, and ultimately improves risk identification capabilities and regulatory effectiveness.

[0224] Fig.15 A schematic diagram of an electronic device 900 suitable for implementing the multimodal large model retrieval method for the security inspection field according to an embodiment of the present disclosure is schematically shown.

[0225] like Fig.15 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0226] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0227] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0228] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0229] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0230] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0231] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0232] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0233] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0234] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0235] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0236] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0237] The embodiments of the present disclosure are described above. However, these embodiments are only for the purpose of illustration and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A multimodal large model retrieval method for security inspection, characterized in that: The method comprises: receiving a user input, the user input comprising a target text and a target image; and Processing the user input according to a search enhancement process; The processing of the user input according to the search enhancement process includes: Extracting an image block to be identified from the target image to obtain at least one first image block; Retrieving the first image block in a security inspection database to obtain a first search result, wherein the first search result includes at least one first security inspection image and first text record data of an image block in the first security inspection image that successfully matches the first image block; wherein the security inspection database is a database formed based on historical security inspection data, and the historical security inspection data includes at least security inspection images and text record data of security inspection images; Inputting the first text record data in the first search result into the large model, and using a preset first prompt word to prompt the large model to determine the type of goods in the first image block according to the input first text information, so as to obtain a first goods recognition result; Obtaining an overall cargo identification result based on a first cargo identification result corresponding to at least one of the first image blocks; Inputting the overall cargo identification result and the target text into the large model, and using a preset second prompt word to prompt the large model to generate a first reply content for the target text according to the overall cargo identification result; and The first reply content is output.

2. The method according to claim 1, characterized in that Processing the user input according to the retrieval enhancement process further includes: using at least one target detection expert model to identify the first image block to obtain a second cargo identification result, wherein the target detection expert model is a machine learning model based on image recognition; Then, obtaining the overall cargo identification result based on the first cargo identification result corresponding to at least one of the first image blocks further includes: obtaining the overall cargo identification result based on the first cargo identification result and the second cargo identification result corresponding to at least one of the first image blocks.

3. The method according to claim 2, characterized in that The at least one target detection expert model includes at least one contraband detection model, wherein each contraband detection model is used to detect a type of contraband.

4. The method according to claim 1, characterized in that The step of extracting the image block to be identified from the target image to obtain at least one first image block comprises: Based on the content of the target text, determine a region to be identified in the target image, wherein the region to be identified includes the entire target image or a user-specified region in the target image; At least one first image block is segmented from the area to be identified based on image texture features.

5. The method according to claim 1, characterized in that The method further comprises: Identifying the user intent of the target text using an intent recognition model; Determining the user intent category based on the user intent and a preset intent classification; When the user intention category is a retrieval enhancement related intention category, the user input is processed according to a retrieval enhancement process, wherein the intention in the retrieval enhancement related intention category includes identifying goods in an image.

6. The method according to claim 5, characterized in that The method further comprises: When the user intention category is a general search intention category, processing the user input according to a general search process, wherein the intention in the general search intention category includes retrieving data from the security inspection database; Wherein, processing the user input according to the general search process includes: Extracting the image block to be retrieved from the target image to obtain at least one second image block; Retrieving the second image block in the security inspection database to obtain a second search result, wherein the second search result includes at least one second security inspection image, and marking information and second text record data of an image block in the second security inspection image that successfully matches the second image block; The preset third prompt word is used to prompt the large model to sort and output the second search result according to the target text.

7. The method according to claim 6, characterized in that The step of searching the security inspection database for the second image block to obtain a second search result further comprises: Preprocessing the target text to determine search scope information; The second image block is searched in the security inspection database according to the search range information to obtain the second search result.

8. The method according to claim 1, characterized in that The security inspection database includes a structured database and an unstructured database, wherein the unstructured database includes at least one of the following: a vector database or a graph database.

9. The method according to claim 1, characterized in that: The target image includes a perspective image, and the security inspection image includes a perspective image.

10. A multi-modal large model retrieval device for security inspection, characterized in that: The device comprises: An image-text interaction module, used to receive user input, wherein the user input includes a target text and a target image; A central scheduling module, used for processing the user input according to the retrieval enhancement process, wherein the central scheduling module processes the user input by calling the image segmentation expert module, the retriever and the retrieval post-processing module; The image segmentation expert module is used to extract the image block to be identified from the target image to obtain at least one first image block; The retriever is used to retrieve the first image block in the security inspection database to obtain a first search result, wherein the first search result includes at least one first security inspection image and first text record data of an image block in the first security inspection image that successfully matches the first image block; wherein the security inspection database is a database formed based on historical security inspection data, and the historical security inspection data at least includes security inspection images and text record data of security inspection images; The post-retrieval processing module is used for: Inputting the first text record data in the first search result into the large model, and using a preset first prompt word to prompt the large model to determine the type of goods in the first image block according to the input first text information, so as to obtain a first goods recognition result; Obtaining an overall cargo identification result based on a first cargo identification result corresponding to at least one of the first image blocks; and Inputting the overall cargo recognition result and the target text into the large model, and using a preset second prompt word to prompt the large model to generate a first reply content for the target text according to the overall cargo recognition result; Among them, the graphic and text interaction module is also used to output the first reply content.

11. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images