Data retrieval method and device, electronic equipment and computer readable storage medium

Abstract

According to the data retrieval method and device, the electronic equipment and the computer readable storage medium provided by the invention, feature extraction is carried out on the related data when the image data is collected to be put in storage and the retrieval request is carried out, so that the matching basis of the data of the image type can be enhanced; therefore, the acquired image data required by the retrieval request is matched based on the obtained data features, the retrieval efficiency can be improved, and meanwhile, the retrieval accuracy of the image type data can be improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a data retrieval method, device, electronic device and computer-readable storage medium. Background Art

[0002] The iteration of intelligent driving solutions has an increasingly strong demand for high-quality data, and how to efficiently produce high-quality data is urgent. In the existing technology, data is usually collected and stored on the vehicle side, and when data is needed, the stored data is retrieved to obtain the required data; however, the retrieval method in the existing technology has low retrieval efficiency, and at the same time, the support for image-type data is poor. Summary of the invention

[0003] The main purpose of the present invention is to propose a data retrieval method, device, electronic device and computer-readable storage medium, aiming to solve the problems of low retrieval efficiency and poor support for image-type data in the prior art.

[0004] To achieve the above object, the present invention provides a data retrieval method, which comprises the steps of:

[0005] Receiving search data and obtaining the data feature extraction model;

[0006] Inputting the search data into the data feature extraction model to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data;

[0007] Matching target data features corresponding to the retrieved data features in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by extracting features from the acquired image data by the feature extraction model;

[0008] The collected image data corresponding to the target data feature is used as a retrieval result.

[0009] Optionally, before matching the target data feature corresponding to the search data feature in the feature database, the method includes:

[0010] Receiving collected image data, inputting the collected image data into the all-target perception large model, and obtaining a first target area in the collected image data;

[0011] Inputting the acquired image data and the first target area into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the first target area in the acquired image data to obtain the corresponding acquired data features of the acquired image data;

[0012] The collected data features are stored in the collected feature library.

[0013] Optionally, the data feature extraction model includes a large model of all-target perception and a large model of a multimodal feature base; the step of inputting the retrieval data into the data feature extraction model to perform feature extraction on the retrieval data through the data feature extraction model to obtain retrieval data features corresponding to the retrieval data includes:

[0014] Determining whether the retrieved data is in image format;

[0015] If the retrieval data is in image format, inputting the retrieval data into the all-target perception large model to obtain a second target area in the retrieval data;

[0016] The retrieval data and the second target area are input into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the second target area in the retrieval data to obtain corresponding retrieval data features of the retrieval data.

[0017] Optionally, the data feature extraction model includes a large model of all-target perception and a large model of a multimodal feature base; the step of inputting the retrieval data into the data feature extraction model to perform feature extraction on the retrieval data through the data feature extraction model to obtain retrieval data features corresponding to the retrieval data includes:

[0018] Determining whether the retrieved data is in image format;

[0019] If the retrieval data is in a non-image format, the retrieval data is input into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the retrieval data to obtain corresponding retrieval data features of the retrieval data.

[0020] Optionally, the step of inputting the search data into the data feature extraction model to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data includes:

[0021] Acquire multiple search sub-data in the search data;

[0022] Inputting each of the search sub-data into the data feature extraction model respectively, so as to extract features of the search sub-data respectively through the data feature extraction model, and obtain search data sub-features corresponding to each of the search sub-data;

[0023] The matching of the target data feature corresponding to the search data feature in the feature database comprises:

[0024] The target data features corresponding to all the retrieved data sub-features are matched in the collected data features.

[0025] Optionally, the step of inputting the search data into the data feature extraction model to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data includes:

[0026] Acquire image retrieval data and text retrieval data from the retrieval data;

[0027] Determine a target retrieval area in the image retrieval data according to the text retrieval data;

[0028] The image retrieval data and the target retrieval area are input into the data feature extraction model, so that the image retrieval data is subjected to feature extraction based on the target retrieval area by the data feature extraction model to obtain retrieval data features corresponding to the retrieval data.

[0029] Optionally, taking the collected image data corresponding to the target data feature as the retrieval result comprises:

[0030] Generate iterative update samples according to the retrieval data, the target data features, and the collected image data;

[0031] The data feature extraction model is updated through the iterative update samples.

[0032] To achieve the above object, the present invention further provides a data retrieval device, the data retrieval device comprising:

[0033] A first receiving module, used for receiving search data and obtaining the data feature extraction model;

[0034] A first input module, used for inputting the search data into the data feature extraction model, so as to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data;

[0035] A first matching module is used to match the target data feature corresponding to the search data feature in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by extracting features from the acquisition image data by the feature extraction model;

[0036] The first execution module is used to use the collected image data corresponding to the target data feature as a retrieval result.

[0037] To achieve the above objectives, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data retrieval method described above when executed by the processor.

[0038] To achieve the above object, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data retrieval method described above are implemented.

[0039] The present invention proposes a data retrieval method, device, electronic device and computer-readable storage medium, which receive retrieval data and obtain the data feature extraction model; input the retrieval data into the data feature extraction model to extract features of the retrieval data through the data feature extraction model to obtain retrieval data features corresponding to the retrieval data; match the target data features corresponding to the retrieval data features in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by the feature extraction model performing feature extraction on the acquired image data; and use the acquired image data corresponding to the target data features as the retrieval result. By extracting features from the relevant data when the acquired image data is stored and when a retrieval request is made, the matching basis for image type data can be strengthened, so that the acquired image data required for the retrieval request can be matched based on the obtained data features, thereby improving the retrieval efficiency and at the same time improving the retrieval accuracy for image type data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 It is a flowchart of the first embodiment of the data retrieval method of the present invention;

[0043] Figure 2 is an overall flow chart of the data retrieval method of the present invention;

[0044] Figure 3 It is a schematic diagram of the structure of the large model of all-target perception in the data retrieval method of the present invention;

[0045] Figure 4 It is a module structure diagram of the data retrieval device of the present invention;

[0046] Figure 5 It is a schematic diagram of the module structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present application.

[0048] The present invention provides a data retrieval method, referring to Figure 1 , Figure 1 This is a flow chart of a first embodiment of a data retrieval method of the present invention, the method comprising the steps of:

[0049] Step S10, receiving the search data and obtaining the data feature extraction model;

[0050] The retrieval data is the relevant information input by the developer when performing retrieval and mining; the retrieval data reflects the characteristics of the target data to be retrieved, that is, it is necessary to retrieve the collected image data that matches the retrieval data; it should be noted that the retrieval data can include input in multiple formats, such as images and text information; it can also be a combination of different formats, such as the retrieval data contains both image and text information.

[0051] After the developer inputs the retrieval data, data retrieval is triggered, and the data feature extraction model is obtained at this time.

[0052] Step S20, inputting the search data into the data feature extraction model, so as to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data;

[0053] It can be understood that when the collected image data is stored in the database, the data feature extraction model is used to extract features of the collected image data to obtain the corresponding collected data features; therefore, when matching, matching can be based on specific data features; therefore, in this embodiment, after obtaining the retrieval data, the data feature extraction model is used to extract features of the retrieval data to obtain the retrieval data features corresponding to the retrieval data; the specific feature extraction can be performed in an analogous manner to the collected image data, and will not be repeated here.

[0054] Step S30, matching the target data features corresponding to the search data features in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by extracting features from the acquired image data by the feature extraction model;

[0055] The collected image data is the image data collected by the vehicle during the actual driving process; the image data can be but not limited to single-view images, multi-view images, video streams, radar point clouds; it can be understood that different types of image data can be collected by setting corresponding collection devices, such as single-view images can be collected by cameras, multi-view images can be collected by multi-eye cameras, and radar point clouds can be collected by radars.

[0056] It is understandable that the collected image data can be obtained from different vehicles. When the user agrees to upload the data, the vehicle uploads the collected image data to the server after collecting the collected image data, and the data retrieval device obtains the collected image data uploaded by the vehicle from the server.

[0057] The data feature extraction model is used to extract features from the collected image data.

[0058] It is understandable that the collected image data is in image format. If the collected image data is directly retrieved based on the image format, the accuracy of the retrieval results is poor; therefore, in order to improve the retrieval accuracy of the collected image data in image format, in this embodiment, the collected image data is feature extracted by a data feature extraction model to obtain the collected data features corresponding to the collected image data; the collected data features reflect the characteristics of the collected image data, and the corresponding collected data features can better reflect its uniqueness compared to the collected image data itself. Therefore, retrieval based on the collected data features can improve the accuracy and efficiency of retrieval for image formats. The specific form of the collected data features can be set based on actual needs, such as feature vectors.

[0059] It can be understood that the collected image data is in image format, and at the same time, the collected image data can also be in different types of image formats; and the retrieval data can also be in different formats; in this embodiment, by extracting features from the collected image data and the retrieval data respectively, it is possible to better integrate features of different modalities and dimensions, thereby reducing the obstacles to retrieval caused by differences in modalities and formats, and improving support for multimodal data.

[0060] Specifically, when the collected image data is stored in the database, the collected data features obtained by feature extraction can be stored in the feature database, and the collected image data can be stored in the collection database. At the same time, there is an association relationship between the corresponding collected data features and the collected image data.

[0061] After the retrieval data feature is obtained, the collected data feature corresponding to the retrieval data feature is matched in the feature database, and the matched collected data feature is used as the target data feature.

[0062] Step S40: taking the collected image data corresponding to the target data feature as a retrieval result.

[0063] After determining the target data features, the collected image data corresponding to the target data features can be determined in the collection database based on the association between the collected data features and the collected image data. The collected image data is then the target data of the retrieved data. Therefore, the collected image data is returned to the developer as a retrieval result to complete the retrieval.

[0064] This embodiment extracts features from relevant data when collecting image data and storing it in the database and when a retrieval request is made, so that the matching basis for image type data can be strengthened, thereby matching the collected image data required for the retrieval request based on the obtained data features, thereby improving the retrieval efficiency and, at the same time, improving the retrieval accuracy for image type data.

[0065] Further, see Figure 2 In the second embodiment of the data retrieval method of the present invention proposed based on the first embodiment of the present invention, the data feature extraction model includes a full-target perception large model and a multi-modal feature base large model; the step before step S30 includes the following steps:

[0066] Step S50, inputting the collected image data into the full-target perception large model to obtain a first target area in the collected image data;

[0067] Step S60, inputting the acquired image data and the first target area into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the first target area in the acquired image data to obtain the corresponding acquired data features of the acquired image data;

[0068] Step S70, storing the collected data features in the collected feature library.

[0069] The large all-target perception model CA-PeLM-4D can support the input of single image, multi-view image, video stream, radar point cloud and other image formats; see Figure 3The full-target perception big model includes a coding module and an MMD-former (Multi Modality and Dimension former) module; after the data is input into the full-target perception big model, it is first encoded in the coding module. At the same time, different encoders can be applied based on the different formats of the data. For example, when the data is a single image, a multi-view image, or a video stream, it can be encoded by an image encoder Image Encoder, and when the data is a radar point cloud, it can be encoded by a point cloud encoder PC Encoder.

[0070] The encoded data is input into the MMD-former module, which fuses multiple modal and multi-dimensional information, such as the interaction between multi-view images, the interaction of continuous video frame sequences in video streams, the interaction between images and radar point clouds, the interaction between perception information and text, etc. At the same time, text can be input into the MMD-former module to achieve full target perception capabilities with the assistance of text input.

[0071] The full-target perception large model ultimately outputs the target area of ​​the data; it can be understood that when collecting image data, the full-target perception large model detects the target area of ​​the collected image data to obtain the first target area; when retrieving data, the full-target perception large model detects the target area of ​​the retrieved data to obtain the second target area.

[0072] The multimodal feature base large model CA-VLfeat-4D includes an encoding module and an LLM (Large Language Model) module; the encoding module of the multimodal feature base large model and the full-target perception large model can be shared; the encoding module interacts with the LLM module through cross queries and adapters; different prompts in the LLM module further align perception and text features; thereby achieving better fusion of features of different modalities and dimensions; at the same time, all tasks are modeled in a unified manner through language sequences.

[0073] It should be noted that the multimodal feature base large model can be set based on the specific image format that needs to be applied; for example, when supporting 2D graphics, a 2D Clip model can be constructed; when supporting multimodal graphics, a multimodal graphics generation large model can be constructed, and when indicating multimodal 3D graphics, a multimodal 3D graphics large model can be constructed; when supporting 4D graphics, a multimodal 4D graphics large model can be constructed; different supported model forms can be implemented by training the multimodal feature base large model through training samples corresponding to specific formats.

[0074] The multimodal base large model finally outputs the data features of the data; it can be understood that, when collecting image data, the multimodal base large model performs feature extraction on the collected image data to obtain the collected data features; when retrieving data, the multimodal base large model performs feature extraction on the retrieval data to obtain the retrieval data features.

[0075] In this embodiment, accurate extraction of data features can be achieved through the full-target perception large model and the multi-modal base large model.

[0076] Further, in a third embodiment of the data retrieval method of the present invention proposed based on the first embodiment of the present invention, the data feature extraction model includes a full-target perception large model and a multimodal feature base large model; the step S20 includes the steps of:

[0077] Step S21, determining whether the retrieved data is in image format;

[0078] Step S22, if the retrieval data is in image format, inputting the retrieval data into the all-target perception large model to obtain a second target area in the retrieval data;

[0079] Step S23, inputting the retrieval data and the second target area into the multimodal feature base macromodel, so that the multimodal feature base macromodel extracts features of the second target area in the retrieval data to obtain corresponding retrieval data features of the retrieval data.

[0080] It can be understood that when the retrieval data is in image format, in order to improve the accuracy of the retrieval, the second target area in the retrieval data is first determined through the full-target perception large model, and then the multimodal feature base large model is used to perform feature extraction based on the second target area to obtain the retrieval feature data; the specific implementation can be performed in an analogous manner to the feature extraction of the aforementioned collected image data, and will not be repeated here.

[0081] Step S24, if the retrieval data is in a non-image format, the retrieval data is input into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the retrieval data to obtain corresponding retrieval data features of the retrieval data.

[0082] When the retrieval data is in a non-image format, such as a text format, there is no need to determine the target area in the retrieval data. Therefore, the retrieval data can be directly input into the multimodal feature base large model for feature extraction.

[0083] In this embodiment, by distinguishing the image format of the retrieval data, when the retrieval data is in image format, the second target area in the retrieval data can be determined through the full target perception large model, and then feature extraction is performed based on the second target area, thereby improving the accuracy of feature extraction and the accuracy of retrieval.

[0084] Further, in a fourth embodiment of the data retrieval method of the present invention proposed based on the first embodiment of the present invention, step S20 includes the steps of:

[0085] Step S25, obtaining a plurality of search sub-data in the search data;

[0086] Step S26, inputting each of the search sub-data into the data feature extraction model respectively, so as to perform feature extraction on the search sub-data respectively through the data feature extraction model, and obtain the search data sub-feature corresponding to each of the search sub-data;

[0087] The step S30 comprises:

[0088] Step S31, matching the target data features corresponding to all the retrieved data sub-features in the collected data features.

[0089] The search data may include one or more search sub-data, and the formats of the multiple search sub-data may be the same or different. For example, the multiple search sub-data may be multiple images or multiple groups of texts, or some of the search sub-data may be images and some may be texts.

[0090] It can be understood that different search sub-data indicate relevant features of the target data. Therefore, feature extraction can be performed on the search sub-data respectively to obtain the search data sub-features corresponding to each search sub-data; it can be understood that multiple search sub-data in the search data indicate the same required target data. Therefore, after determining the search data sub-features, the target feature data corresponding to all the search sub-data are matched in the feature database.

[0091] It should be noted that when the collected image data is stored in the database, feature extraction of the collected image data can obtain collection feature data containing multiple corresponding collection feature sub-data, that is, the multiple collection feature sub-data are all associated with the collected image data; similarly, a single retrieval sub-data in the retrieval data can also obtain multiple corresponding retrieval feature sub-data through feature extraction.

[0092] In this embodiment, the target data features are matched by combining multiple search feature sub-data, so that the search accuracy can be improved.

[0093] Further, in a fifth embodiment of the data retrieval method of the present invention proposed based on the first embodiment of the present invention, step S20 includes the steps of:

[0094] Step S29, obtaining image retrieval data and text retrieval data in the retrieval data;

[0095] Step S210, determining a target search area in the image search data according to the text search data;

[0096] Step S211, inputting the image retrieval data and the target retrieval area into the data feature extraction model, so as to extract features of the image retrieval data based on the target retrieval area through the data feature extraction model to obtain retrieval data features corresponding to the retrieval data.

[0097] When the retrieval data contains both image and text formats, the text retrieval data can be used to assist in determining the target retrieval area in the image retrieval data. Specifically, the description of the text retrieval data is used to find the area related to the description in the image retrieval data to determine the target retrieval area, thereby improving the accuracy of feature extraction of the retrieval data.

[0098] After determining the target retrieval area, the image retrieval data and the target retrieval area can be input into the data feature extraction model, so as to perform feature extraction on the image retrieval data based on the target retrieval area.

[0099] It should be noted that the determination of the target retrieval area in the present embodiment can be implemented by an independently set module; it can also be implemented by a data feature extraction model, such as inputting the image retrieval data and the text retrieval data into the full target perception large model in the data feature extraction model. The full target perception large model can determine the target retrieval area in the image retrieval data based on the text retrieval data through multimodal data fusion.

[0100] Furthermore, in a sixth embodiment of the data retrieval method of the present invention proposed based on the first embodiment of the present invention, the step S40 includes the following steps:

[0101] Step S80, generating an iterative update sample according to the search data, the target data features, and the collected image data;

[0102] Step S90: updating the data feature extraction model through the iterative update sample.

[0103] After each retrieval is completed, the target data features described in the retrieval data and the corresponding collected image data and other data can be obtained; in this embodiment, the data feature extraction model is continuously iteratively updated through the data generated by the retrieval. It can be understood that it can be specifically iteratively updated in the full target perception large model and the multimodal feature base large model.

[0104] It is understandable that the input of the data feature extraction model is data and the output is data features, so the data and data features generated by the retrieval can be used to generate iterative update samples. The specific implementation method of updating the model through iterative update samples can be set based on actual needs and will not be repeated here.

[0105] It should be noted that preference settings can be made in the selection of iterative update samples; for example, after the retrieval is completed, determine whether the data involved in the retrieval is long-tail / difficult data; long-tail / difficult data refers to data that occurs in fewer scenarios or has more retrieval judgment errors. For this type of data, the retrieval results are often less accurate due to the small number of samples during the retrieval process. Therefore, for long-tail / difficult data, the model needs to be continuously iteratively updated based on this type of data.

[0106] After the retrieval is completed, developers can annotate the data generated by the retrieval, such as the type of annotated data, whether the retrieval results are correct, etc. Based on the annotated content, the type of data can be determined and positive or negative samples can be generated. For unlabeled data, sub-annotation and data cleaning can be performed through the model to achieve high data quality, which facilitates continuous iteration and upgrade optimization of the model.

[0107] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0109] The present application also provides a data retrieval device for implementing the above data retrieval method. Figure 4 , the data retrieval device comprises:

[0110] A first receiving module, used for receiving search data and obtaining the data feature extraction model;

[0111] A first input module, used for inputting the search data into the data feature extraction model, so as to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data;

[0112] A first matching module is used to match the target data feature corresponding to the search data feature in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by extracting features from the acquisition image data by the feature extraction model;

[0113] The first execution module is used to use the collected image data corresponding to the target data feature as a retrieval result.

[0114] The data retrieval device extracts features from relevant data when collecting image data and storing it in the database and when a retrieval request is made, so as to strengthen the matching basis for image type data, thereby matching the collected image data required for the retrieval request based on the obtained data features, thereby improving the retrieval efficiency and, at the same time, improving the retrieval accuracy for image type data.

[0115] It should be noted that the first receiving module in this embodiment can be used to execute step S10 in the embodiment of the present application, the first input module in this embodiment can be used to execute step S20 in the embodiment of the present application, the first matching module in this embodiment can be used to execute step S30 in the embodiment of the present application, and the first execution module in this embodiment can be used to execute step S40 in the embodiment of the present application.

[0116] Furthermore, the data feature extraction model includes a large model of all-target perception and a large model of multimodal feature base; the device includes:

[0117] A first input module, used for inputting the collected image data into the all-target perception large model to obtain a first target area in the collected image data;

[0118] A second input module is used to input the acquired image data and the first target area into the multimodal feature base large model, so that the multimodal feature base large model extracts features of the first target area in the acquired image data to obtain the corresponding acquired data features of the acquired image data;

[0119] The first storage module is used to store the collected data features in the collected feature library.

[0120] Furthermore, the data feature extraction model includes a large model of all-target perception and a large model of multimodal feature base; the second input module includes:

[0121] A first judging unit, used to judge whether the retrieved data is in image format;

[0122] A first input unit, configured to input the retrieval data into the all-target perception large model if the retrieval data is in an image format, to obtain a second target region in the retrieval data;

[0123] The second input unit is used to input the retrieval data and the second target area into the multimodal feature base large model, so that the multimodal feature base large model extracts features of the second target area in the retrieval data to obtain corresponding retrieval data features of the retrieval data.

[0124] Furthermore, the data feature extraction model includes a large model of all-target perception and a large model of multimodal feature base; the second input module includes:

[0125] A first judging unit, used to judge whether the retrieved data is in image format;

[0126] The third input unit is used to input the retrieval data into the multimodal feature base large model if the retrieval data is in a non-image format, so that the multimodal feature base large model performs feature extraction on the retrieval data to obtain corresponding retrieval data features of the retrieval data.

[0127] Furthermore, the second input module includes:

[0128] A first acquisition unit, used for acquiring a plurality of search sub-data in the search data;

[0129] A fourth input unit, used for inputting each of the search sub-data into the data feature extraction model, so as to perform feature extraction on the search sub-data through the data feature extraction model, and obtain search data sub-features corresponding to each of the search sub-data;

[0130] The first matching module comprises:

[0131] The first matching unit is used to match the target data features corresponding to all the retrieved data sub-features in the collected data features.

[0132] Furthermore, the second input module includes:

[0133] A second acquisition unit, used to acquire image retrieval data and text retrieval data from the retrieval data;

[0134] A first determining unit, configured to determine a target retrieval area in the image retrieval data according to the text retrieval data;

[0135] The fifth input unit is used to input the image retrieval data and the target retrieval area into the data feature extraction model, so as to perform feature extraction on the image retrieval data based on the target retrieval area through the data feature extraction model to obtain retrieval data features corresponding to the retrieval data.

[0136] Furthermore, the device also includes:

[0137] A first generating module, used for generating iterative update samples according to the search data, the target data features, and the collected image data;

[0138] The first updating module is used to update the data feature extraction model through the iterative update sample.

[0139] Reference Figure 5 In terms of hardware structure, the electronic device may include components such as a communication module 10, a memory 20, and a processor 30. In the electronic device, the processor 30 is connected to the memory 20 and the communication module 10 respectively, and a computer program is stored in the memory 20. The computer program is executed by the processor 30 at the same time, and the steps of the above method embodiment are implemented when the computer program is executed.

[0140] The communication module 10 can be connected to an external communication device through a network. The communication module 10 can receive requests from an external communication device, and can also send requests, instructions and information to the external communication device, which can be other electronic devices, servers or Internet of Things devices, such as televisions, etc.

[0141] The memory 20 can be used to store software programs and various data. The memory 20 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as receiving and retrieving data), etc.; the data storage area can include a database, and the data storage area can store data or information created according to the use of the system, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0142] The processor 30 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, so as to monitor the electronic device as a whole. The processor 30 may include one or more processing units; optionally, the processor 30 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 30.

[0143] although Figure 5 Although not shown, the electronic device may further include a circuit control module, which is used to connect to a power source to ensure the normal operation of other components. Figure 5 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0144] The present invention also provides a computer-readable storage medium on which a computer program is stored. The computer-readable storage medium may be Figure 5 The memory 20 in the electronic device may also be at least one of a ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, and an optical disk. The computer-readable storage medium includes a number of instructions for enabling a terminal device with a processor (which may be a television, a car, a mobile phone, a computer, a server, a terminal, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0145] In the present invention, the terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0147] Although the embodiments of the present invention have been shown and described above, the scope of protection of the present invention is not limited thereto. It is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify and replace the above embodiments within the scope of the present invention, and these changes, modifications and replacements should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A data retrieval method, characterized in that: The data retrieval method comprises: Receiving search data and obtaining the data feature extraction model; Inputting the search data into the data feature extraction model to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data; Matching target data features corresponding to the retrieved data features in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by extracting features from the acquired image data by the feature extraction model; The collected image data corresponding to the target data feature is used as a retrieval result.

2. The data retrieval method according to claim 1, characterized in that: The data feature extraction model includes a large model of all-target perception and a large model of multi-modal feature base; Before matching the target data feature corresponding to the search data feature in the feature database, the method includes: Receiving collected image data, inputting the collected image data into the all-target perception large model, and obtaining a first target area in the collected image data; Inputting the acquired image data and the first target area into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the first target area in the acquired image data to obtain the corresponding acquired data features of the acquired image data; The collected data features are stored in the collected feature library.

3. The data retrieval method according to claim 1, characterized in that: The data feature extraction model includes a large model of all-target perception and a large model of a multimodal feature base; the step of inputting the retrieval data into the data feature extraction model to extract features of the retrieval data through the data feature extraction model, and obtaining retrieval data features corresponding to the retrieval data includes: Determining whether the retrieved data is in image format; If the retrieval data is in image format, inputting the retrieval data into the all-target perception large model to obtain a second target area in the retrieval data; The retrieval data and the second target area are input into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the second target area in the retrieval data to obtain corresponding retrieval data features of the retrieval data.

4. The data retrieval method according to claim 1, characterized in that: The data feature extraction model includes a large model of all-target perception and a large model of a multimodal feature base; the step of inputting the retrieval data into the data feature extraction model to extract features of the retrieval data through the data feature extraction model, and obtaining retrieval data features corresponding to the retrieval data includes: Determining whether the retrieved data is in image format; If the retrieval data is in a non-image format, the retrieval data is input into the multimodal feature base macromodel, so that the multimodal feature base macromodel performs feature extraction on the retrieval data to obtain corresponding retrieval data features of the retrieval data.

5. The data retrieval method according to claim 1, characterized in that: The step of inputting the search data into the data feature extraction model to extract features of the search data through the data feature extraction model to obtain search data features corresponding to the search data includes: Acquire multiple search sub-data in the search data; Inputting each of the search sub-data into the data feature extraction model respectively, so as to extract features of the search sub-data respectively through the data feature extraction model, and obtain search data sub-features corresponding to each of the search sub-data; The matching of the target data feature corresponding to the search data feature in the feature database comprises: The target data features corresponding to all the retrieved data sub-features are matched in the collected data features.

6. The data retrieval method according to claim 1, characterized in that: The step of inputting the search data into the data feature extraction model to extract features of the search data through the data feature extraction model to obtain search data features corresponding to the search data includes: Acquire image retrieval data and text retrieval data from the retrieval data; Determine a target retrieval area in the image retrieval data according to the text retrieval data; The image retrieval data and the target retrieval area are input into the data feature extraction model, so that the image retrieval data is subjected to feature extraction based on the target retrieval area by the data feature extraction model to obtain retrieval data features corresponding to the retrieval data.

7. The data retrieval method according to claim 1, characterized in that: The step of taking the collected image data corresponding to the target data feature as the retrieval result includes: Generate iterative update samples according to the retrieval data, the target data features, and the collected image data; The data feature extraction model is updated through the iterative update samples.

8. A data retrieval device, characterized in that: The data retrieval device comprises: A first receiving module, used for receiving search data and obtaining the data feature extraction model; A first input module, used for inputting the search data into the data feature extraction model, so as to perform feature extraction on the search data through the data feature extraction model to obtain search data features corresponding to the search data; A first matching module is used to match the target data feature corresponding to the search data feature in a feature database, wherein the feature database includes acquisition feature data, and the acquisition feature data is obtained by extracting features from the acquisition image data by the feature extraction model; The first execution module is used to use the collected image data corresponding to the target data feature as a retrieval result.

9. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data retrieval method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data retrieval method according to any one of claims 1 to 7 are implemented.

Images