Data mining method and device based on multi-modal collaboration

Abstract

The invention provides a data mining method and device based on multi-modal collaboration.The method comprises the steps that retrieval is conducted, specifically, a data retrieval request is received, retrieval is conducted according to retrieval data in the data retrieval request, and a current retrieval result is obtained and stored in a retrieval request result set; a termination judgment step: performing termination condition judgment according to the retrieval request result set; a multi-modal retrieval step: under the condition that the retrieval request result set does not meet the termination condition, performing multi-modal retrieval to obtain a multi-modal retrieval result; and a repeating step: mining associated data according to the multi-modal retrieval result and the retrieval request result set, repeatedly executing the retrieval step, the termination judgment step, the multi-modal retrieval step and the repeating step at least once until the retrieval request result set meets a termination condition, and outputting the retrieval request result set. According to the method and the device, the problem that in the prior art, retrieval is performed by depending on single-modal information according to a retrieval request, and associated data retrieval is not performed, so that the retrieval accuracy is reduced is solved.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and in particular to a data mining method based on multimodal collaboration, a data mining device based on multimodal collaboration, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the popularization of the Internet, the rise of big data and cloud computing, and the advancement of technologies such as artificial intelligence and deep learning, the continuous innovation and rapid development of intelligent video analysis technology have been jointly promoted. Among them, data retrieval is an important branch of intelligent video analysis. Data retrieval is not only of great significance for in-process and post-process data analysis in different industries such as transportation and public security; at the same time, with the rapid development of smart cities and mobile Internet, the scale of data has also shown exponential growth. How to efficiently, accurately and maximize the retrieval of data that meets the requirements from massive amounts of disordered data has a positive effect on further promoting the advancement of technologies such as artificial intelligence; especially for long-tail tasks where data collection is difficult, an efficient data retrieval system can greatly speed up the development and iteration of models.

[0003] In recent years, multimodal models can simultaneously contain multiple different types of data and understand and interpret these data from different perspectives. For example, the Contrastive Language-Image Pre-training (CLIP) model maps text and image features into the same space to achieve alignment between text and images, which enables cross-modal learning and cross-modal retrieval capabilities. Based on CLIP, information in one modality (such as text, image, etc.) can be retrieved from another modality (such as image, video, etc.) with similar semantics.

[0004] However, the existing methods still have the following problems: (1) The current retrieval system does not have the ability to mine related data based on the retrieval results and automatically search based on the related data; using a text-image search example, for example, when the retrieval data "a person holding a child wearing a mask" is input, conventional retrieval cannot mine "a person holding a child wearing a blue mask" or "a person holding a child wearing a white mask" and other related fine-grained data as new retrieval data based on the retrieval results, and use this as a new retrieval input for retrieval, while iterating this logic and automatically determining whether to terminate the process, so as to maximize the retrieval of data that meets the request from the database. (2) Due to factors such as model training data and model preference, single-modal data and its features have problems such as information uniformity and insufficient information understanding and expression capabilities. (3) Traditional feature engineering analysis methods (such as but not limited to clustering) are difficult to capture the deep semantic information and association relationships expressed in feature data. Summary of the invention

[0005] The main purpose of the present application is to provide a data mining method based on multimodal collaboration, a data mining device based on multimodal collaboration, a computer-readable storage medium and a computer program product, so as to at least solve the problem in the prior art that retrieval accuracy is reduced by relying on single modal information to perform retrieval based on a retrieval request without performing associated data retrieval.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data mining method based on multimodal collaboration is provided, including: a retrieval step, receiving a retrieval data request, and searching according to the retrieval data in the retrieval data request to obtain a current retrieval result, and storing the current retrieval result in a retrieval request result set, wherein the retrieval data request at least includes the initiation time of the client's request and the retrieval data, and the retrieval request result set includes multiple retrieval results; a termination judgment step, performing a termination condition judgment according to the retrieval request result set, and the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the retrieval results meet the set total number of retrievals. The number of iterations reaches the set maximum number of iterations, and the number of retrieval iterations is the total number of retrievals performed according to the retrieval data request; a multimodal retrieval step, when the retrieval request result set does not meet the termination condition, a multimodal retrieval is performed to obtain a multimodal retrieval result; a repetition step, mining related data according to the multimodal retrieval result and the retrieval request result set, and receiving a new retrieval data request initiated according to the related data, and repeating the retrieval step, the termination judgment step, the multimodal retrieval step and the repetition step in sequence at least once until the retrieval request result set meets the termination condition, and outputting the retrieval request result set to the client.

[0007] Optionally, the retrieval data is one of the following: first retrieval data and second retrieval data, the first retrieval data is the target retrieval keyword issued by the client when performing the first retrieval, and the second retrieval data is the retrieval result in the associated data that is not repeated in the retrieval request result set. After receiving the retrieval data request, the method further includes: determining whether the first retrieval data exists in the retrieval data request; if the first retrieval data does not exist in the retrieval data request, issuing a first prompt message and terminating the retrieval, the first prompt message being used to prompt that there is no valid retrieval data in the retrieval data request issued by the client and that retrieval cannot be performed; if the first retrieval data exists in the retrieval data request, performing retrieval according to the first retrieval data in the retrieval data request.

[0008] Optionally, a termination condition is judged based on the retrieval request result set, including: judging whether the total number of retrieval results in the retrieval request result set is greater than or equal to the set total retrieval number; when the total number is greater than or equal to the set total retrieval number, judging that the retrieval request result set satisfies the termination condition and terminating the retrieval; when the amount of retrieval result data is less than the set total retrieval number, judging whether the number of retrieval iterations has reached the set maximum number of iterations; when the number of retrieval iterations has reached the set maximum number of iterations, judging that the retrieval request result set satisfies the termination condition and terminating the retrieval; when the number of retrieval iterations has not reached the set maximum number of iterations, performing a multimodal retrieval to obtain a multimodal retrieval result.

[0009] Optionally, after performing a multimodal retrieval and obtaining a multimodal retrieval result, the method further includes: calculating the similarity between each of the multimodal retrieval results and the retrieval data request to obtain a plurality of multimodal retrieval similarities; calculating an average value based on all the multimodal retrieval similarities to obtain a similarity average value; when the similarity average value is greater than a set evaluation threshold, increasing the set maximum number of iterations; when the similarity average value is less than or equal to the set evaluation threshold, reducing the set maximum number of iterations.

[0010] Optionally, the retrieval request result set also includes a similarity corresponding to each of the retrieval results, where the similarity is the degree of similarity between the retrieval result and the retrieval data request. Before mining related data based on the multimodal retrieval results and the retrieval request result set, the method also includes: when the multimodal retrieval result has a latest retrieval result, adding the latest retrieval result to the retrieval request result set, where the latest retrieval result is a retrieval result in the multimodal retrieval result that is not repeated in the retrieval request result set; when the multimodal retrieval result is repeated with the retrieval result stored in the retrieval request result set, weighted addition is performed on the similarity of the multimodal retrieval result and the similarity of the repeated retrieval result, and the similarity corresponding to the repeated retrieval result is updated, where the repeated retrieval result is the retrieval result in the retrieval request result set that is repeated with the multimodal retrieval result.

[0011] Optionally, the retrieval request result set also includes a similarity corresponding to each of the retrieval results, wherein the similarity is the degree of similarity between the retrieval result and the retrieval data request, and mining related data based on the multimodal retrieval result and the retrieval request result set, including: merging the multimodal retrieval result and the retrieval request result set for deduplication processing, and updating the retrieval request result set, wherein the deduplication processing is used to remove duplicate retrieval results; selecting the first set number of retrieval results in the retrieval request result set in descending order of the similarity to obtain related retrieval results; performing feature extraction, clustering processing and feature enhancement processing on the related retrieval results in sequence to obtain enhanced features, and simultaneously performing semantic enhancement processing on the related retrieval results using a visual language algorithm to obtain semantic features; constructing collaborative features based on the enhanced features and the semantic features in combination with a first formula, wherein the first formula is f 1 represents the enhancement feature, f 2 represents the semantic feature, F s represents the synergistic feature, Indicates the similarity between the enhanced feature and the semantic feature; inputs the collaborative feature and the first search data into a similarity measurement network, and outputs a sorting result, wherein the first search data is a target search keyword issued by the client when performing the first search, and the sorting result is a result obtained by sorting the similarity between the collaborative feature and the first search data from large to small; selects the collaborative features before the second set number according to the sorting result to obtain the associated data.

[0012] Optionally, after receiving a new retrieval data request initiated based on the associated data, the method further includes: determining whether there is second retrieval data in the associated data, the second retrieval data being a retrieval result in the associated data that is not repeated with the retrieval request result set; in a case where the second retrieval result does not exist in the associated data, issuing a second prompt message and terminating the retrieval, the second prompt message being used to prompt that the second retrieval data does not exist in the associated data; in a case where the second retrieval result exists in the associated data, repeating the retrieval step, the termination judgment step, the multimodal retrieval step and the repetition step at least once until the retrieval request result set meets the termination condition.

[0013] According to another aspect of the present application, a data mining device based on multimodal collaboration is provided, the device comprising: a first retrieval unit, used to execute a retrieval step, receive a retrieval data request, and perform a search according to the retrieval data in the retrieval data request to obtain a current retrieval result, and store the current retrieval result in a retrieval request result set, wherein the retrieval data request at least includes the initiation request time of the client and the retrieval data, and the retrieval request result set includes multiple retrieval results; a termination judgment unit, used to execute a termination judgment step, and perform a termination condition judgment according to the retrieval request result set, wherein the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the number of retrieval iterations is greater than or equal to the number of retrieval iterations. the number of retrieval iterations reaches a set maximum number of iterations, the number of retrieval iterations being the total number of retrievals performed according to the retrieval data request; a multimodal retrieval unit, for executing a multimodal retrieval step, and performing a multimodal retrieval to obtain a multimodal retrieval result when the retrieval request result set does not satisfy the termination condition; a first repeating unit, for executing a repeating step, mining associated data according to the multimodal retrieval result and the retrieval request result set, receiving a new retrieval data request initiated according to the associated data, and sequentially repeating the retrieval step, the termination judgment step, the multimodal retrieval step and the repeating step at least once until the retrieval request result set satisfies the termination condition, and outputting the retrieval request result set to the client.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0015] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the methods described above is implemented.

[0016] By applying the technical solution of the present application, in a data mining method based on multimodal collaboration, first, in a retrieval step, a retrieval data request is received, and a retrieval is performed according to the retrieval data in the retrieval data request to obtain a current retrieval result, and the current retrieval result is stored in a retrieval request result set, wherein the retrieval data request at least includes the client's initiation request time and the retrieval data, and the retrieval request result set includes multiple retrieval results; then, in a termination judgment step, a termination condition judgment is performed according to the retrieval request result set, and the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the number of retrieval iterations reaches a set number The maximum number of iterations is determined, and the above-mentioned number of retrieval iterations is the total number of retrievals performed according to the above-mentioned retrieval data request; thereafter, in the multimodal retrieval step, when the above-mentioned retrieval request result set does not meet the above-mentioned termination condition, a multimodal retrieval is performed to obtain a multimodal retrieval result; finally, the repeating step is performed to mine related data according to the above-mentioned multimodal retrieval result and the above-mentioned retrieval request result set, and a new retrieval data request initiated according to the above-mentioned related data is received, and the above-mentioned retrieval step, the above-mentioned termination judgment step, the above-mentioned multimodal retrieval step and the repeating step are sequentially repeated at least once until the above-mentioned retrieval request result set meets the above-mentioned termination condition, and the above-mentioned retrieval request result set is output to the above-mentioned client. The present application first obtains retrieval data according to the retrieval data request for retrieval; then a termination judgment is performed according to all the retrieval results (i.e., the above-mentioned retrieval request result set), and if the termination condition is met, the retrieval is terminated and the retrieval request result is output; if the termination condition is not met, a multimodal retrieval is performed; then related data mining is performed according to the multimodal retrieval result and the retrieval request result set, and finally the related data obtained by the related data mining is used as a new retrieval request for re-retrieval. The selection of related data fully utilizes the cross-modal understanding and generation capabilities of the multimodal large model, overcomes the problem of insufficient expression of single modal information, and combines with the proposed feature enhancement, collaborative feature construction and multimodal collaborative similarity measurement network to accurately select related data as new retrieval requests; termination judgment is an adaptive termination judgment method, which automatically updates the termination parameters according to the retrieval results, so as to achieve the purpose of efficiently and accurately retrieving data that meets the requirements from massive data. This application solves the problem in the prior art that the retrieval accuracy is reduced by relying on single modal information for retrieval according to the retrieval request without performing related data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a data mining method based on multimodal collaboration provided in an embodiment of the present application is shown;

[0018] Figure 2A flowchart of a data mining method based on multimodal collaboration provided according to an embodiment of the present application is shown;

[0019] Figure 3 A flowchart of a data mining method based on multimodal collaboration is shown according to an embodiment of the present application;

[0020] Figure 4 A flowchart of an adaptive termination judgment provided according to an embodiment of the present application is shown;

[0021] Figure 5 A flowchart of selecting associated data provided according to an embodiment of the present application is shown;

[0022] Figure 6 A structural block diagram of a data mining device based on multimodal collaboration provided according to an embodiment of the present application is shown.

[0023] The above drawings include the following reference numerals:

[0024] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] As introduced in the background technology, the current retrieval system in the prior art does not have the ability to mine related data based on the retrieval results and automatically search based on the related data, and due to factors such as model training data and model preference, single-modal data and its characteristics have problems such as information uniformity, insufficient information understanding and expression capabilities, etc. In order to solve the problem that the prior art relies on single-modal information to search according to retrieval requests without searching for related data, resulting in reduced retrieval accuracy, the embodiments of the present application provide a data mining method based on multimodal collaboration, a data mining device based on multimodal collaboration, a computer-readable storage medium and a computer program product.

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal for executing a data mining method based on multimodal collaboration according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data mining method based on multimodal collaboration in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a data mining method based on multimodal collaboration is provided, which runs on a mobile terminal, a computer terminal or a similar computing device. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 is a flow chart of a data mining method based on multimodal collaboration according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0034] Step S201, a retrieval step, receives a retrieval data request, and performs a search based on the retrieval data in the retrieval data request to obtain a current retrieval result, and stores the current retrieval result in a retrieval request result set. The retrieval data request at least includes the client's initiation request time and the retrieval data, and the retrieval request result set includes multiple retrieval results.

[0035] Specifically, the system receives a request for retrieval data from a client, and the request contains at least the time when the request was initiated and the retrieval data (such as text, image, etc.). Based on the retrieval data in the request, the system starts the retrieval process and searches the database through a multimodal large model to find the data points most relevant to the retrieval data. The current result obtained from the retrieval is stored in the retrieval request result set, which continuously accumulates the results of each retrieval.

[0036] Step S202, the termination judgment step, performs termination condition judgment based on the above-mentioned search request result set, the above-mentioned termination condition includes that the total number of the above-mentioned search results in the above-mentioned search request result set meets the set total number of searches, or the number of search iterations reaches the set maximum number of iterations, and the above-mentioned number of search iterations is the total number of searches performed according to the above-mentioned search data request.

[0037] Specifically, Figure 3 As shown, the system checks whether the search request result set meets the preset termination conditions, including whether the total number of search results reaches the set total number of searches, or whether the current number of search iterations reaches the set maximum number of iterations. If any of the termination conditions is met, the system determines that the search process can be terminated; otherwise, the search process continues. By setting the termination conditions, the system can avoid unnecessary repeated searches, save computing resources and time, and improve the efficiency of the entire system.

[0038] Step S203, a multimodal search step, in which, when the above search request result set does not satisfy the above termination condition, a multimodal search is performed to obtain a multimodal search result.

[0039] Specifically, when the result set of the search request does not meet the termination condition, the system extracts multimodal features from the current search results and uses a large multimodal model to deeply understand data in different modalities such as text and images. Based on feature enhancement and semantic information generated by the VLM (Vision Language Model) model, the system constructs collaborative features to more comprehensively characterize the correlation and semantic richness between data. The system uses the constructed collaborative features to perform multimodal retrieval, mine more potential related data, and further enrich the retrieval results. The multimodal retrieval step uses a large multimodal model to deeply understand the data, improving the semantic richness and expressiveness of the retrieval results.

[0040] Step S204, repeating steps, mining related data according to the above multimodal retrieval results and the above retrieval request result set, and receiving a new retrieval data request initiated according to the above related data, repeating the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repeating step in sequence at least once until the above retrieval request result set meets the above termination condition, and outputting the above retrieval request result set to the above client.

[0041] Specifically, based on the multimodal retrieval results, the system mines out the data associated with the retrieval request result set. Based on the mined associated data, a new retrieval data request is generated for the next round of retrieval. Based on the new retrieval data request, the system re-executes the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repetition step until the retrieval request result set meets the termination condition. Once the termination condition is met, the system outputs the final retrieval request result set to the client. Through loop iteration, the data related to the retrieval request in the database can be deeply mined to ensure the depth and breadth of the retrieval. By receiving retrieval requests, intelligently judging the termination of retrieval, executing multimodal retrieval and loop iteration, the system can effectively control the retrieval process and avoid waste of resources. At the same time, through deep data mining and result expansion, the accuracy and recall rate of retrieval are improved, providing a high-quality data foundation for subsequent data analysis or model training.

[0042] In this embodiment, in the data mining method based on multimodal collaboration, first, in a retrieval step, a retrieval data request is received, and a retrieval is performed according to the retrieval data in the retrieval data request to obtain a current retrieval result, and the current retrieval result is stored in a retrieval request result set, wherein the retrieval data request at least includes the client's initiation request time and the retrieval data, and the retrieval request result set includes multiple retrieval results; then, in a termination judgment step, a termination condition judgment is performed according to the retrieval request result set, and the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the number of retrieval iterations reaches the set maximum The number of iterations, the above-mentioned number of retrieval iterations is the total number of retrievals performed according to the above-mentioned retrieval data request; thereafter, in the multimodal retrieval step, when the above-mentioned retrieval request result set does not meet the above-mentioned termination condition, a multimodal retrieval is performed to obtain a multimodal retrieval result; finally, a repeating step is performed to mine related data according to the above-mentioned multimodal retrieval results and the above-mentioned retrieval request result set, and a new retrieval data request initiated according to the above-mentioned related data is received, and the above-mentioned retrieval step, the above-mentioned termination judgment step, the above-mentioned multimodal retrieval step and the repeating step are sequentially repeated at least once until the above-mentioned retrieval request result set meets the above-mentioned termination condition, and the above-mentioned retrieval request result set is output to the above-mentioned client. The present application first obtains retrieval data according to the retrieval data request for retrieval; then a termination judgment is performed according to all retrieval results (i.e., the above-mentioned retrieval request result set), and if the termination condition is met, the retrieval is terminated and the retrieval request result is output; if the termination condition is not met, a multimodal retrieval is performed; then related data mining is performed according to the multimodal retrieval results and the retrieval request result set, and finally the related data obtained by the related data mining is used as a new retrieval request for re-retrieval. The selection of related data fully utilizes the cross-modal understanding and generation capabilities of the multimodal large model, overcomes the problem of insufficient expression of single modal information, and combines with the proposed feature enhancement, collaborative feature construction and multimodal collaborative similarity measurement network to accurately select related data as new retrieval requests; termination judgment is an adaptive termination judgment method, which automatically updates the termination parameters according to the retrieval results, so as to achieve the purpose of efficiently and accurately retrieving data that meets the requirements from massive data. This application solves the problem in the prior art that the retrieval accuracy is reduced by relying on single modal information for retrieval according to the retrieval request without performing related data retrieval.

[0043] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the data mining method based on multimodal collaboration of the present application will be described in detail below in combination with specific embodiments.

[0044] In order to avoid invalid retrieval and improve resource utilization efficiency, in an optional implementation manner, after the above step S201, the method further includes:

[0045] Step S301, determining whether the first search data exists in the search data request;

[0046] Step S302, when the first search data does not exist in the search data request, a first prompt message is issued and the search is terminated, wherein the first prompt message is used to prompt that the search data request issued by the client does not contain valid search data and cannot be searched;

[0047] Step S303: When the first search data exists in the search data request, search is performed according to the first search data in the search data request.

[0048] In the above embodiment, the above retrieval data is one of the following: first retrieval data and second retrieval data, the above first retrieval data is the target retrieval keyword issued by the client when the first retrieval is performed, and the above second retrieval data is the retrieval result in the above associated data that is not repeated with the above retrieval request result set. The above is the pre-check and startup stage of the retrieval process, which ensures the legitimacy and validity of the retrieval request, thereby avoiding meaningless computing consumption and improving the response efficiency of the system and user experience. First, the retrieval data request issued by the client is parsed to identify the key information therein, that is, the above first retrieval data. This can be a text description, image, video clip, etc. input by the user. The system checks whether the above first retrieval data exists in the request, that is, whether the request contains valid retrieval information, rather than an empty request or a request with an incorrect format. By checking the existence of the first retrieval data, the system can effectively avoid invalid retrieval caused by an empty request or an incorrect format, saving computing resources. Timely feedback of request errors, sending a first prompt message to the client, informing the user that there is a lack of valid retrieval data in the request, helps to improve the user experience and avoid unnecessary troubles for the user in the process of waiting for fruitless retrieval. If it is determined that the first search data does not exist in the search request, a first prompt message is generated, which clearly indicates that there is no valid search data in the request. No subsequent search processing will be performed on invalid or empty requests, but the search process will be terminated directly to avoid resource waste. Figure 4 As shown, when there is no search request data in the search data request, the search is terminated directly. Once the system confirms that the search request contains valid first search data, these data are used as the basis for the search. The search process is started according to the first search data, and the multimodal data most relevant to the search data is mined from the database through a multimodal large model and a collaborative feature construction mechanism. Ensure that the search process is started based on valid search data, and be able to accurately retrieve information that best matches user needs from the database, thereby improving the accuracy of the search. Start the search immediately after receiving a valid search request, which reduces unnecessary waiting time, speeds up the system response speed, and improves the overall processing efficiency.

[0049] In order to avoid excessive retrieval and data processing, in an optional implementation manner, the above step S202 includes:

[0050] Step S2021, determining whether the total number of the search results in the search request result set is greater than or equal to the set total number of searches;

[0051] Step S2022, when the total number is greater than or equal to the set total number of searches, determining that the search request result set meets the termination condition, and terminating the search;

[0052] Step S2023, when the amount of the search result data is less than the set total number of searches, determining whether the number of search iterations reaches the set maximum number of iterations;

[0053] Step S2024: When the number of search iterations reaches the set maximum number of iterations, it is determined that the search request result set meets the termination condition, and the search is terminated.

[0054] In the above embodiment, an adaptive termination judgment method based on the combination of random forest and association analysis technology is proposed. This method can reduce the dependence on artificial parameters and automatically increase or decrease the number of iterative retrievals according to the retrieval results to achieve the purpose of high efficiency and maximized retrieval data; it constitutes a termination judgment mechanism in the retrieval process, and realizes intelligent control of the retrieval process by dynamically evaluating the number of retrieval results and the number of iterations, thereby preventing excessive retrieval and waste of resources and ensuring that the retrieval is both efficient and sufficient. Specifically, Figure 4As shown, the total number of search results in the current search request result set is counted in real time, including all the data results that have been retrieved. The total number of search results counted is compared with the preset total number of searches to determine whether the preset search scale has been reached. By setting the total number of searches, the system can control the scale of the search, avoid data processing and transmission bottlenecks caused by too many search results, and ensure that data search is completed within a reasonable time and resource consumption range. At the same time, it can also ensure that the search results reach a certain number, which can improve the coverage of the search, so that the system can mine as much data associated with the search request as possible, especially when processing long-tail tasks, it can fully collect rare or low-frequency data information. If it is determined that the total number of search results has met or exceeded the set total number of searches, the system will determine that the search request result set meets the termination condition. The system immediately stops further search operations and no longer initiates new search requests or performs data mining. The system tracks the number of iterations that have been executed during the search process and compares it with the set maximum number of iterations. If the current number of iterations has not reached the set maximum value, the system will continue to execute the search process to mine more relevant data. By setting the maximum number of iterations, the system can avoid falling into an infinite loop of search and ensure that the search process does not consume excessive resources. The system checks whether the current number of retrieval iterations is equal to the set maximum number of iterations. If the number of iterations has reached the maximum value, the system determines that the termination condition is met and immediately stops the retrieval process without performing new multimodal retrieval or associated data mining. By limiting the number of iterations, the retrieval process runs within the preset range, improving the controllability and predictability of the process, avoiding excessive retrieval and data processing, and reducing computing costs and storage pressure.

[0055] In order to optimize the efficiency and effect of data retrieval, in an optional implementation manner, after the above step S203, the method further includes:

[0056] Step S401, calculating the similarity between each of the multimodal search results and the search data request to obtain a plurality of multimodal search similarities;

[0057] Step S402, calculating an average value based on all the above multimodal retrieval similarities to obtain an average similarity value;

[0058] Step S403, when the similarity average value is greater than the set evaluation threshold, increase the set maximum number of iterations;

[0059] Step S404: when the similarity average value is less than or equal to the set evaluation threshold, reduce the set maximum number of iterations.

[0060] In the above embodiment, the average similarity of all retrieval results is calculated based on the results and their similarity with the retrieval request. If the average value is greater than the evaluation threshold, it means that the retrieval result is good, and then the number of iterations is updated; if the average value is less than the evaluation threshold, it means that the retrieval result is poor, and then the number of iterations is reduced, and the process is terminated in time. The number of retrieval iterations is dynamically adjusted according to the semantic similarity of the retrieval results, thereby optimizing the efficiency and effect of data retrieval. The specific detailed process is: First, the system uses a multimodal large model (such as CLIP, VLM, etc.) to extract features from the retrieval request and the multimodal retrieval results, and maps data of different modalities such as text and images to a unified feature space. Then, through the similarity measurement network, based on the extracted features, the semantic similarity between the retrieval results and the original retrieval data request is calculated. This step can use the attention mechanism and the Transformer architecture to capture the deep association between features, so as to more accurately measure the degree of semantic matching. The calculated similarity values ​​between each multimodal retrieval result and the retrieval data request are summarized. The summarized similarity values ​​are averaged to obtain a similarity average value reflecting the quality of the overall retrieval results. The average similarity provides a quantitative indicator for evaluating the overall quality of the current search result set and whether it is highly relevant to the search request. Figure 4 As shown, the obtained average similarity value is compared with the preset evaluation threshold. If the average similarity value is higher than this evaluation threshold, it means that the semantic match of the current search result is high, and the system decides to increase the maximum number of search iterations in order to further improve the coverage and diversity of the search results. Increasing the number of iterations allows for deeper searches, mining more data that matches the search criteria, and further improving the quality and coverage of the data set. However, when the average similarity value is lower than or equal to the evaluation threshold, it means that the current search result has a low match with the search request, and the system will reduce the maximum number of iterations to avoid unnecessary consumption of computing resources. When the quality of the search results declines, the system can automatically adjust the strategy and terminate the search process in time to avoid invalid repeated searches and ensure the efficiency and accuracy of the entire search process.

[0061] In order to reduce redundant data, in an optional implementation manner, before the above step S204, the method further includes:

[0062] Step S501, if there is a latest search result in the multimodal search result, add the latest search result to the search request result set, where the latest search result is a search result in the multimodal search result that is not repeated in the search request result set;

[0063] Step S502, when the above-mentioned multimodal retrieval result is repeated with the above-mentioned retrieval result stored in the above-mentioned retrieval request result set, the similarity of the above-mentioned multimodal retrieval result and the similarity of the repeated retrieval result are weightedly added, and the above-mentioned similarity corresponding to the above-mentioned repeated retrieval result is updated. The above-mentioned repeated retrieval result is the above-mentioned retrieval result in the above-mentioned retrieval request result set that is repeated with the above-mentioned multimodal retrieval result.

[0064] In the above embodiment, the above retrieval request result set also includes the similarity corresponding to each of the above retrieval results, and the above similarity is the similarity between the above retrieval result and the above retrieval data request. In the process of re-searching according to the newly mined related data, each iterative search will have retrieval results that are repeated with the historical retrieval data, that is, there are retrieval results that are repeated with the retrieval request result set. When this happens, two strategies for updating the retrieval data in the retrieval request result set are adopted, which not only reduces redundant data, but also ensures the weight update of the newly added retrieval result data and the historical retrieval result data. The first case is when there are new search results in the multimodal search results, that is, when they are not repeated with the historical search data in the search request result set, the multimodal search results are added to the search request result set. For example, assuming that there are 100 search result data in the search request result set, when the current multimodal search result retrieves 50 search result data, these 50 search result data are compared with the 100 search result data in the search request result set, and 20 duplicate search result data are found. Therefore, the remaining 30 search result data are added to the search request result set. At this time, a total of 130 search result data are stored in the search request result set; the second case is if it is found that there are new search results in the latest round of multimodal search results. If there are retrieval results that are duplicates of the data in the retrieval request result set, these retrieval results are marked as duplicate retrieval results. Then, when the multimodal retrieval results are duplicates of the historical retrieval data, these duplicate retrieval results are not updated, but the similarities of these duplicate retrieval results are updated. The similarities are updated by weighted addition of the similarities of the multimodal retrieval results and the corresponding duplicate retrieval results. The above example is still used to illustrate that after it is found that the multimodal retrieval results have 20 duplicate retrieval result data, the similarities of these 20 retrieval result data in the multimodal retrieval results are calculated and weighted addition is performed with the similarities of the duplicate retrieval results stored in the retrieval request result set to obtain a new similarity, and the similarities of these 20 retrieval result data in the retrieval request result set are updated.

[0065] In order to improve the accuracy of associated data mining, in an optional implementation, the above step S204 includes:

[0066] Step S2041, merging the multimodal search results and the search request result set to remove duplicates, and updating the search request result set, wherein the deduplication process is used to remove duplicate search results;

[0067] Step S2042, selecting the first set number of the search results in the search request result set in descending order of the similarity, to obtain associated search results;

[0068] Step S2043, performing feature extraction, clustering processing and feature enhancement processing on the above-mentioned related search results in sequence to obtain enhanced features, and at the same time using a visual language algorithm to perform semantic enhancement processing on the above-mentioned related search results to obtain semantic features;

[0069] Step S2044, constructing collaborative features based on the enhanced features and the semantic features combined with the first formula, the first formula is: f 1 represents the above enhanced features, f 2 Represents the above semantic features, F s represents the above synergistic features, Indicates the similarity between the above enhanced features and the above semantic features;

[0070] Step S2045, inputting the collaborative feature and the first search data into a similarity measurement network, and outputting a ranking result, wherein the first search data is a target search keyword sent by the client when performing the first search, and the ranking result is a result obtained by ranking the similarity between the collaborative feature and the first search data from large to small;

[0071] Step S2046: selecting the collaborative features of the first second set number according to the sorting result to obtain the associated data.

[0072] In the above embodiment, the above retrieval request result set also includes the similarity corresponding to each of the above retrieval results, and the above similarity is the similarity between the above retrieval result and the above retrieval data request. Figure 5 A flowchart of selecting related data according to an embodiment of the present application is shown. The related data selection method mainly includes two stages: online retrieval process and offline model training. Figure 5 As shown in Figure 2, the online retrieval process improves the accuracy of associated data mining through feature enhancement, collaborative feature construction and similarity measurement network. Figure 5The single search result in () and the above search request result set are merged and deduplicated, and the above search request result set is updated. The features obtained by feature extraction only represent the current data features, and do not contain the relationship between classes within a class, resulting in low discrimination between features and difficulty in mining associated data. Therefore, after obtaining the central features of N clusters through the clustering algorithm, they are added to the original features in the cluster for feature enhancement to obtain enhanced features, which increases the discrimination between classes within a class. The calculation formula for feature enhancement is as follows: Where: {i|i∈N, 1≤i≤N} represents N clusters, i represents the cluster number; {j|j∈M, 1≤j≤M} represents that there are M features in each cluster, j represents the feature number, C i represents the central feature of the i-th cluster, represents the jth original feature in the i-th cluster, represents the jth enhanced feature in the i-th cluster, and the set of all enhanced features is denoted as f 1 .

[0073] The features obtained by feature enhancement are unimodal features, which have problems such as insufficient semantic expression. Therefore, in the process of feature construction, a visual language model (VLM) (such as but not limited to Interpretable Vision-Language, InternVL and other multimodal models) is introduced to generate semantic information using its modal understanding and generation capabilities to obtain semantic features, denoted as f 2 , which makes up for the problem of insufficient expression of single-modal feature information. The collaborative feature construction is shown in the first formula above. It can dynamically construct collaborative features according to the correlation between the two features. When the similarity between the two features is low, it relies more on enhanced features. When the similarity between the two features is high, more semantic information is introduced. In this way, the original feature information can be retained, and semantic features can be introduced to ensure feature quality.

[0074] Generally speaking, the features extracted by multimodal models have richer semantic expression capabilities. However, traditional feature engineering analysis methods (such as clustering, residual similarity, etc.) are difficult to capture the deep semantic information and association relationships expressed in the feature data. Therefore, a transformer-based similarity measurement network is proposed. The similarity measurement network uses the attention mechanism to capture the deep information and association relationships in the feature data, and calculates the softmax activation function according to the data results of the Linear layer in the transformer model, and then re-sorts them, that is, the similarity between the collaborative feature vector and the retrieval data request is sorted from large to small, and the top K data (that is, the second set number mentioned above) are selected as the above-mentioned associated data.

[0075] It should be noted that offline training of the similarity measurement network is mainly based on annotated data to train a similarity measurement model that automatically captures semantic information and associations between features. The main steps are as follows: Step 1) Collect and save the features constructed by collaborative features during the online retrieval process and the original image information corresponding to the features, and then manually score the image based on the retrieval data request and the original image information; Step 2) Train the similarity measurement network based on the results of step 1), and the loss function loss is calculated as the regression loss loss 1 and sorting loss 2 Weighted Sum in, loss 2 (x1,x2)=max(0,(x1-x2)+margin), where is a hyperparameter used to coordinate the weights of different losses, y i Indicates the actual value, Represents the square of the predicted value, x1 represents the first input data of any two sample input data, x2 represents the other input data of any two sample input data, and the default value of margin is 0.

[0076] In order to ensure the diversity and novelty of the search results and avoid unnecessary consumption of resources, in an optional implementation, after the above step S204, the method further includes:

[0077] Step S601, determining whether there is second search data in the associated data, the second search data being a search result in the associated data that is not duplicated with the search request result set;

[0078] Step S602, when the second search result does not exist in the associated data, a second prompt message is issued and the search is terminated, wherein the second prompt message is used to prompt that the second search data does not exist in the associated data;

[0079] Step S603, when the second search result exists in the associated data, repeat the search step, the termination judgment step, the multimodal search step and the repetition step at least once until the search request result set satisfies the termination condition.

[0080] In the above embodiment, the focus is on the decision to screen and re-retrieve the mined associated data, ensuring the diversity and novelty of the search results, while also avoiding unnecessary consumption of resources. The mined associated data are analyzed, and these data are potential related results obtained after multimodal retrieval based on feature enhancement, collaborative feature construction and similarity measurement network. The system identifies the retrieval results in the associated data that are not repeated with the current search request result set, that is, the second retrieval data, to ensure that the mined new data can bring additional value to the result set. By identifying the second retrieval data, the system can avoid incorporating repeated information into the retrieval result set again, thereby improving the purity and efficiency of the data set. Ensuring that the second retrieval data contained in the associated data is new information helps to expand the scope of the search and discover more data that has not been retrieved before. If it is found that there is no second retrieval data in the associated data, that is, all data are repeated with the existing results, the system will generate a second prompt message. Stop further retrieval and data mining operations based on the second prompt message to prevent meaningless repeated retrieval from wasting resources. If the second search data is identified in the associated data, the system will use these data as new search inputs. If the second search result exists in the associated data, the system will repeat the search step, the termination judgment step, the multimodal search step and the repetition step at least once until the search request result set meets the termination condition. After each iteration, the system will update the search request result set and add the newly mined second search data. By continuously using the second search data for re-search, the system can continuously mine the undiscovered associated data in the database, deepen the breadth and depth of data retrieval, and use the dynamic iteration mechanism until the termination condition is met, which ensures the adequacy of the search process and avoids premature termination and missing potential valuable data. In the iterative process, not only the data deduplication and novelty of the search result set are ensured, but also the balance between effective resource utilization and search adequacy is achieved through dynamic iteration and adaptive termination mechanism. The overall performance of the data mining method and retrieval system based on multimodal collaboration is significantly improved, especially in the scenario of processing large-scale, multimodal data, which can better cope with the dynamic changes of data and the diversity of retrieval requirements, and provide a high-quality data foundation for subsequent analysis and application.

[0081] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0082] The embodiment of the present application also provides a data mining device based on multimodal collaboration. It should be noted that the data mining device based on multimodal collaboration of the embodiment of the present application can be used to execute the data mining method based on multimodal collaboration provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0083] The following is an introduction to the data mining device based on multimodal collaboration provided in an embodiment of the present application.

[0084] Figure 6 is a structural block diagram of a data mining device based on multimodal collaboration according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0085] The first retrieval unit 10 is used to execute the retrieval step, receive a retrieval data request, and perform a search based on the retrieval data in the retrieval data request to obtain a current retrieval result, and store the current retrieval result in a retrieval request result set. The retrieval data request at least includes the client's initiation request time and the retrieval data. The retrieval request result set includes multiple retrieval results.

[0086] Specifically, the system receives a request for retrieval data from a client, and the request contains at least the time when the request was initiated and the retrieval data (such as text, image, etc.). Based on the retrieval data in the request, the system starts the retrieval process and searches the database through a multimodal large model to find the data points most relevant to the retrieval data. The current result obtained from the retrieval is stored in the retrieval request result set, which continuously accumulates the results of each retrieval.

[0087] The termination judgment unit 20 is used to execute the termination judgment step and perform termination condition judgment based on the above-mentioned search request result set. The above-mentioned termination condition includes that the total number of the above-mentioned search results in the above-mentioned search request result set meets the set total number of searches, or the number of search iterations reaches the set maximum number of iterations. The above-mentioned number of search iterations is the total number of searches performed according to the above-mentioned search data request.

[0088] Specifically, Figure 3As shown, the system checks whether the search request result set meets the preset termination conditions, including whether the total number of search results reaches the set total number of searches, or whether the current number of search iterations reaches the set maximum number of iterations. If any of the termination conditions is met, the system determines that the search process can be terminated; otherwise, the search process continues. By setting the termination conditions, the system can avoid unnecessary repeated searches, save computing resources and time, and improve the efficiency of the entire system.

[0089] The multimodal retrieval unit 30 is used to execute the multimodal retrieval step, and when the above-mentioned retrieval request result set does not meet the above-mentioned termination condition, perform a multimodal retrieval to obtain a multimodal retrieval result.

[0090] Specifically, when the result set of the search request does not meet the termination condition, the system extracts multimodal features from the current search results and uses a large multimodal model to deeply understand data in different modalities such as text and images. Based on feature enhancement and semantic information generated by the VLM (Vision Language Model) model, the system constructs collaborative features to more comprehensively characterize the correlation and semantic richness between data. The system uses the constructed collaborative features to perform multimodal retrieval, mine more potential related data, and further enrich the retrieval results. The multimodal retrieval step uses a large multimodal model to deeply understand the data, improving the semantic richness and expressiveness of the retrieval results.

[0091] The first repeating unit 40 is used to execute the repetitive step, mine related data according to the above multimodal retrieval results and the above retrieval request result set, and receive a new retrieval data request initiated according to the above related data, and repeat the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repetitive step in sequence at least once until the above retrieval request result set meets the above termination condition, and output the above retrieval request result set to the above client.

[0092] Specifically, based on the multimodal retrieval results, the system mines out the data associated with the retrieval request result set. Based on the mined associated data, a new retrieval data request is generated for the next round of retrieval. Based on the new retrieval data request, the system re-executes the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repetition step until the retrieval request result set meets the termination condition. Once the termination condition is met, the system outputs the final retrieval request result set to the client. Through loop iteration, the data related to the retrieval request in the database can be deeply mined to ensure the depth and breadth of the retrieval. By receiving retrieval requests, intelligently judging the termination of retrieval, executing multimodal retrieval and loop iteration, the system can effectively control the retrieval process and avoid waste of resources. At the same time, through deep data mining and result expansion, the accuracy and recall rate of retrieval are improved, providing a high-quality data foundation for subsequent data analysis or model training.

[0093] In this embodiment, a first retrieval unit is used to execute a retrieval step, receive a retrieval data request, and perform a search according to the retrieval data in the retrieval data request to obtain a current retrieval result, and store the current retrieval result in a retrieval request result set, wherein the retrieval data request at least includes the initiation time of the request by the client and the retrieval data, and the retrieval request result set includes multiple retrieval results; a termination judgment unit is used to execute a termination judgment step, and perform a termination condition judgment according to the retrieval request result set, wherein the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the number of retrieval iterations reaches the set maximum number of iterations, and the retrieval The number of iterations is the total number of retrievals performed according to the above retrieval data request; a multimodal retrieval unit is used to execute the multimodal retrieval step, and when the above retrieval request result set does not meet the above termination condition, a multimodal retrieval is performed to obtain a multimodal retrieval result; a first repetition unit is used to execute the repetition step, mine related data according to the above multimodal retrieval result and the above retrieval request result set, and receive a new retrieval data request initiated according to the above related data, and repeat the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repetition step at least once in sequence until the above retrieval request result set meets the above termination condition, and output the above retrieval request result set to the above client. The present application first obtains retrieval data according to the retrieval data request for retrieval; then a termination judgment is performed according to all retrieval results (i.e., the above retrieval request result set), and if the termination condition is met, the retrieval is terminated and the retrieval request result is output; if the termination condition is not met, a multimodal retrieval is performed; then related data mining is performed according to the multimodal retrieval result and the retrieval request result set, and finally the related data obtained by the related data mining is used as a new retrieval request for re-retrieval. The selection of related data fully utilizes the cross-modal understanding and generation capabilities of the multimodal large model, overcomes the problem of insufficient expression of single modal information, and combines with the proposed feature enhancement, collaborative feature construction and multimodal collaborative similarity measurement network to accurately select related data as new retrieval requests; termination judgment is an adaptive termination judgment method, which automatically updates the termination parameters according to the retrieval results, so as to achieve the purpose of efficiently and accurately retrieving data that meets the requirements from massive data. This application solves the problem in the prior art that the retrieval accuracy is reduced by relying on single modal information for retrieval according to the retrieval request without performing related data retrieval.

[0094] In order to avoid invalid retrieval and improve resource utilization efficiency, in an optional implementation manner, the device further includes:

[0095] A first determination unit, configured to determine whether the first search data exists in the search data request after receiving the search data request;

[0096] A first prompt termination unit, configured to issue a first prompt message and terminate the search when the first search data does not exist in the search data request, wherein the first prompt message is used to prompt the client that the search data request issued by the client does not contain valid search data and cannot be searched;

[0097] The second retrieval unit is used to perform retrieval according to the first retrieval data in the retrieval data request when the first retrieval data exists in the retrieval data request.

[0098] In the above embodiment, the above retrieval data is one of the following: first retrieval data and second retrieval data, the above first retrieval data is the target retrieval keyword issued by the client when the first retrieval is performed, and the above second retrieval data is the retrieval result in the above associated data that is not repeated with the above retrieval request result set. The above is the pre-check and startup stage of the retrieval process, which ensures the legitimacy and validity of the retrieval request, thereby avoiding meaningless computing consumption and improving the response efficiency of the system and user experience. First, the retrieval data request issued by the client is parsed to identify the key information therein, that is, the above first retrieval data. This can be a text description, image, video clip, etc. input by the user. The system checks whether the above first retrieval data exists in the request, that is, whether the request contains valid retrieval information, rather than an empty request or a request with an incorrect format. By checking the existence of the first retrieval data, the system can effectively avoid invalid retrieval caused by an empty request or an incorrect format, saving computing resources. Timely feedback of request errors, sending a first prompt message to the client, informing the user that there is a lack of valid retrieval data in the request, helps to improve the user experience and avoid unnecessary troubles for the user in the process of waiting for fruitless retrieval. If it is determined that the first search data does not exist in the search request, a first prompt message is generated, which clearly indicates that there is no valid search data in the request. No subsequent search processing will be performed on invalid or empty requests, but the search process will be terminated directly to avoid resource waste. Figure 4 As shown, when there is no search request data in the search data request, the search is terminated directly. Once the system confirms that the search request contains valid first search data, these data are used as the basis for the search. The search process is started according to the first search data, and the multimodal data most relevant to the search data is mined from the database through a multimodal large model and a collaborative feature construction mechanism. Ensure that the search process is started based on valid search data, and be able to accurately retrieve information that best matches user needs from the database, thereby improving the accuracy of the search. Start the search immediately after receiving a valid search request, which reduces unnecessary waiting time, speeds up the system response speed, and improves the overall processing efficiency.

[0099] In order to avoid excessive retrieval and data processing, in an optional implementation manner, the termination judgment unit includes:

[0100] A first determination module is used to determine whether the total number of the search results in the search request result set is greater than or equal to the set total number of searches;

[0101] A first determination module is used to determine that the search request result set meets the termination condition and terminate the search when the total number is greater than or equal to the set total number of searches;

[0102] A second judgment module is used to judge whether the number of search iterations reaches the set maximum number of iterations when the amount of search result data is less than the set total number of searches;

[0103] The second determination module is used to determine whether the search request result set meets the termination condition and terminate the search when the number of search iterations reaches the set maximum number of iterations.

[0104] In the above embodiment, an adaptive termination judgment method based on the combination of random forest and association analysis technology is proposed. This method can reduce the dependence on artificial parameters and automatically increase or decrease the number of iterative retrievals according to the retrieval results to achieve the purpose of high efficiency and maximized retrieval data; it constitutes a termination judgment mechanism in the retrieval process, and realizes intelligent control of the retrieval process by dynamically evaluating the number of retrieval results and the number of iterations, thereby preventing excessive retrieval and waste of resources and ensuring that the retrieval is both efficient and sufficient. Specifically, Figure 4As shown, the total number of search results in the current search request result set is counted in real time, including all the data results that have been retrieved. The total number of search results counted is compared with the preset total number of searches to determine whether the preset search scale has been reached. By setting the total number of searches, the system can control the scale of the search, avoid data processing and transmission bottlenecks caused by too many search results, and ensure that data search is completed within a reasonable time and resource consumption range. At the same time, it can also ensure that the search results reach a certain number, which can improve the coverage of the search, so that the system can mine as much data associated with the search request as possible, especially when processing long-tail tasks, it can fully collect rare or low-frequency data information. If it is determined that the total number of search results has met or exceeded the set total number of searches, the system will determine that the search request result set meets the termination condition. The system immediately stops further search operations and no longer initiates new search requests or performs data mining. The system tracks the number of iterations that have been executed during the search process and compares it with the set maximum number of iterations. If the current number of iterations has not reached the set maximum value, the system will continue to execute the search process to mine more relevant data. By setting the maximum number of iterations, the system can avoid falling into an infinite loop of search and ensure that the search process does not consume excessive resources. The system checks whether the current number of retrieval iterations is equal to the set maximum number of iterations. If the number of iterations has reached the maximum value, the system determines that the termination condition is met and immediately stops the retrieval process without performing new multimodal retrieval or associated data mining. By limiting the number of iterations, the retrieval process runs within the preset range, improving the controllability and predictability of the process, avoiding excessive retrieval and data processing, and reducing computing costs and storage pressure.

[0105] In order to optimize the efficiency and effect of data retrieval, in an optional implementation manner, the device further includes:

[0106] A first calculation unit is used to calculate the similarity between each of the multimodal search results and the search data request after performing a multimodal search and obtaining a multimodal search result, so as to obtain a plurality of multimodal search similarities;

[0107] A second calculation unit is used to calculate an average value based on all the multimodal retrieval similarities to obtain an average similarity value;

[0108] An increasing unit, used for increasing the set maximum number of iterations when the similarity average value is greater than a set evaluation threshold;

[0109] The reducing unit is used to reduce the set maximum number of iterations when the similarity average value is less than or equal to the set evaluation threshold.

[0110] In the above embodiment, the average similarity of all retrieval results is calculated based on the results and their similarity with the retrieval request. If the average value is greater than the evaluation threshold, it means that the retrieval result is good, and then the number of iterations is updated; if the average value is less than the evaluation threshold, it means that the retrieval result is poor, and then the number of iterations is reduced, and the process is terminated in time. The number of retrieval iterations is dynamically adjusted according to the semantic similarity of the retrieval results, thereby optimizing the efficiency and effect of data retrieval. The specific detailed process is: First, the system uses a multimodal large model (such as CLIP, VLM, etc.) to extract features from the retrieval request and the multimodal retrieval results, and maps data of different modalities such as text and images to a unified feature space. Then, through the similarity measurement network, based on the extracted features, the semantic similarity between the retrieval results and the original retrieval data request is calculated. This step can use the attention mechanism and the Transformer architecture to capture the deep association between features, so as to more accurately measure the degree of semantic matching. The calculated similarity values ​​between each multimodal retrieval result and the retrieval data request are summarized. The summarized similarity values ​​are averaged to obtain a similarity average value reflecting the quality of the overall retrieval results. The average similarity provides a quantitative indicator for evaluating the overall quality of the current search result set and whether it is highly relevant to the search request. Figure 4 As shown, the obtained average similarity value is compared with the preset evaluation threshold. If the average similarity value is higher than this evaluation threshold, it means that the semantic match of the current search result is high, and the system decides to increase the maximum number of search iterations in order to further improve the coverage and diversity of the search results. Increasing the number of iterations allows for deeper searches, mining more data that matches the search criteria, and further improving the quality and coverage of the data set. However, when the average similarity value is lower than or equal to the evaluation threshold, it means that the current search result has a low match with the search request, and the system will reduce the maximum number of iterations to avoid unnecessary consumption of computing resources. When the quality of the search results declines, the system can automatically adjust the strategy and terminate the search process in time to avoid invalid repeated searches and ensure the efficiency and accuracy of the entire search process.

[0111] In order to reduce redundant data, in an optional implementation manner, the device further includes:

[0112] A first updating unit is configured to, before mining related data based on the multimodal search results and the search request result set, add the latest search results to the search request result set if there are latest search results in the multimodal search results, wherein the latest search results are search results in the multimodal search results that are not duplicated with the search request result set;

[0113] The second updating unit is used to perform weighted addition of the similarity of the multimodal retrieval result and the similarity of the repeated retrieval result when the multimodal retrieval result is repeated with the retrieval result stored in the retrieval request result set, and update the similarity corresponding to the repeated retrieval result. The repeated retrieval result is the retrieval result in the retrieval request result set that is repeated with the multimodal retrieval result.

[0114] In the above embodiment, the above retrieval request result set also includes the similarity corresponding to each of the above retrieval results, and the above similarity is the similarity between the above retrieval result and the above retrieval data request. In the process of re-searching according to the newly mined related data, each iterative search will have retrieval results that are repeated with the historical retrieval data, that is, there are retrieval results that are repeated with the retrieval request result set. When this happens, two strategies for updating the retrieval data in the retrieval request result set are adopted, which not only reduces redundant data, but also ensures the weight update of the newly added retrieval result data and the historical retrieval result data. The first case is when there are new search results in the multimodal search results, that is, when they are not repeated with the historical search data in the search request result set, the multimodal search results are added to the search request result set. For example, assuming that there are 100 search result data in the search request result set, when the current multimodal search result retrieves 50 search result data, these 50 search result data are compared with the 100 search result data in the search request result set, and 20 duplicate search result data are found. Therefore, the remaining 30 search result data are added to the search request result set. At this time, a total of 130 search result data are stored in the search request result set; the second case is if it is found that there are new search results in the latest round of multimodal search results. If there are retrieval results that are duplicates of the data in the retrieval request result set, these retrieval results are marked as duplicate retrieval results. Then, when the multimodal retrieval results are duplicates of the historical retrieval data, these duplicate retrieval results are not updated, but the similarities of these duplicate retrieval results are updated. The similarities are updated by weighted addition of the similarities of the multimodal retrieval results and the corresponding duplicate retrieval results. The above example is still used to illustrate that after it is found that the multimodal retrieval results have 20 duplicate retrieval result data, the similarities of these 20 retrieval result data in the multimodal retrieval results are calculated and weighted addition is performed with the similarities of the duplicate retrieval results stored in the retrieval request result set to obtain a new similarity, and the similarities of these 20 retrieval result data in the retrieval request result set are updated.

[0115] In order to improve the accuracy of association data mining, in an optional implementation, the first repeating unit includes:

[0116] An updating module, configured to merge the multimodal search results and the search request result set to perform deduplication processing, and update the search request result set, wherein the deduplication processing is used to remove duplicate search results;

[0117] A first selection module is used to select a first set number of the search results in the search request result set in descending order of the similarity to obtain related search results;

[0118] A feature enhancement module is used to perform feature extraction, clustering and feature enhancement processing on the above-mentioned related search results in sequence to obtain enhanced features, and at the same time use a visual language algorithm to perform semantic enhancement processing on the above-mentioned related search results to obtain semantic features;

[0119] A feature construction module is used to construct a collaborative feature based on the enhanced feature and the semantic feature combined with a first formula. The first formula is: f 1 represents the above enhanced features, f 2 Represents the above semantic features, F s represents the above synergistic features, Indicates the similarity between the above enhanced features and the above semantic features;

[0120] An output module is used to input the collaborative feature and the first search data into a similarity measurement network, and output a ranking result, wherein the first search data is a target search keyword sent by the client when performing the first search, and the ranking result is a result obtained by sorting the similarity between the collaborative feature and the first search data from large to small;

[0121] The second selection module is used to select the collaborative features of the second set number before the sorting result to obtain the associated data.

[0122] In the above embodiment, the above retrieval request result set also includes the similarity corresponding to each of the above retrieval results, and the above similarity is the similarity between the above retrieval result and the above retrieval data request. Figure 5 A flowchart of selecting related data according to an embodiment of the present application is shown. The related data selection method mainly includes two stages: online retrieval process and offline model training. Figure 5 As shown in Figure 2, the online retrieval process improves the accuracy of associated data mining through feature enhancement, collaborative feature construction and similarity measurement network. Figure 5The single search result in () and the above search request result set are merged and deduplicated, and the above search request result set is updated. The features obtained by feature extraction only represent the current data features, and do not contain the relationship between classes within a class, resulting in low discrimination between features and difficulty in mining associated data. Therefore, after obtaining the central features of N clusters through the clustering algorithm, they are added to the original features in the cluster for feature enhancement to obtain enhanced features, which increases the discrimination between classes within a class. The calculation formula for feature enhancement is as follows: Where: {i|i∈N, 1≤i≤N} represents N clusters, i represents the cluster number; {j|j∈M, 1≤j≤M} represents that there are M features in each cluster, j represents the feature number, C i represents the central feature of the i-th cluster, represents the jth original feature in the i-th cluster, represents the jth enhanced feature in the i-th cluster, and the set of all enhanced features is denoted as f 1 .

[0123] The features obtained by feature enhancement are unimodal features, which have problems such as insufficient semantic expression. Therefore, in the process of feature construction, a visual language model (VLM) (such as but not limited to Interpretable Vision-Language, InternVL and other multimodal models) is introduced to generate semantic information using its modal understanding and generation capabilities to obtain semantic features, denoted as f 2 , which makes up for the problem of insufficient expression of single-modal feature information. The collaborative feature construction is shown in the first formula above. It can dynamically construct collaborative features according to the correlation between the two features. When the similarity between the two features is low, it relies more on enhanced features. When the similarity between the two features is high, more semantic information is introduced. In this way, the original feature information can be retained, and semantic features can be introduced to ensure feature quality.

[0124] Generally speaking, the features extracted by multimodal models have richer semantic expression capabilities. However, traditional feature engineering analysis methods (such as clustering, residual similarity, etc.) are difficult to capture the deep semantic information and association relationships expressed in the feature data. Therefore, a transformer-based similarity measurement network is proposed. The similarity measurement network uses the attention mechanism to capture the deep information and association relationships in the feature data, and calculates the softmax activation function according to the data results of the Linear layer in the transformer model, and then re-sorts them, that is, the similarity between the collaborative feature vector and the retrieval data request is sorted from large to small, and the top K data (that is, the second set number mentioned above) are selected as the above-mentioned associated data.

[0125] It should be noted that offline training of the similarity measurement network is mainly based on annotated data to train a similarity measurement model that automatically captures semantic information and associations between features. The main steps are as follows: Step 1) Collect and save the features constructed by collaborative features during the online retrieval process and the original image information corresponding to the features, and then manually score the image based on the retrieval data request and the original image information; Step 2) Train the similarity measurement network based on the results of step 1), and the loss function loss is calculated as the regression loss loss 1 and sorting loss 2 Weighted Sum in, loss 2 (x1,x2)=max(0,(x1-x2)+margin), where is a hyperparameter used to coordinate the weights of different losses, y i Indicates the actual value, Represents the square of the predicted value, x1 represents the first input data of any two sample input data, x2 represents the other input data of any two sample input data, and the default value of margin is 0.

[0126] In order to ensure the diversity and novelty of the search results and avoid unnecessary consumption of resources, in an optional implementation manner, the device further includes:

[0127] A second determination unit is configured to determine whether there is second search data in the associated data after receiving a new search data request initiated according to the associated data, wherein the second search data is a search result in the associated data that is not repeated with the search request result set;

[0128] A second prompt termination unit, configured to issue a second prompt message and terminate the search when the second search result does not exist in the associated data, wherein the second prompt message is used to prompt that the second search data does not exist in the associated data;

[0129] The second repeating unit is used to repeat the above-mentioned retrieval step, the above-mentioned termination judgment step, the above-mentioned multimodal retrieval step and the repeating step at least once until the above-mentioned retrieval request result set satisfies the above-mentioned termination condition when the above-mentioned second retrieval result exists in the above-mentioned associated data.

[0130] In the above embodiment, the focus is on the decision to screen and re-retrieve the mined associated data, ensuring the diversity and novelty of the search results, while also avoiding unnecessary consumption of resources. The mined associated data are analyzed, and these data are potential related results obtained after multimodal retrieval based on feature enhancement, collaborative feature construction and similarity measurement network. The system identifies the retrieval results in the associated data that are not repeated with the current search request result set, that is, the second retrieval data, to ensure that the mined new data can bring additional value to the result set. By identifying the second retrieval data, the system can avoid incorporating repeated information into the retrieval result set again, thereby improving the purity and efficiency of the data set. Ensuring that the second retrieval data contained in the associated data is new information helps to expand the scope of the search and discover more data that has not been retrieved before. If it is found that there is no second retrieval data in the associated data, that is, all data are repeated with the existing results, the system will generate a second prompt message. Stop further retrieval and data mining operations based on the second prompt message to prevent meaningless repeated retrieval from wasting resources. If the second search data is identified in the associated data, the system will use these data as new search inputs. If the second search result exists in the associated data, the system will repeat the search step, the termination judgment step, the multimodal search step and the repetition step at least once until the search request result set meets the termination condition. After each iteration, the system will update the search request result set and add the newly mined second search data. By continuously using the second search data for re-search, the system can continuously mine the undiscovered associated data in the database, deepen the breadth and depth of data retrieval, and use the dynamic iteration mechanism until the termination condition is met, which ensures the adequacy of the search process and avoids premature termination and missing potential valuable data. In the iterative process, not only the data deduplication and novelty of the search result set are ensured, but also the balance between effective resource utilization and search adequacy is achieved through dynamic iteration and adaptive termination mechanism. The overall performance of the data mining method and retrieval system based on multimodal collaboration is significantly improved, especially in the scenario of processing large-scale, multimodal data, which can better cope with the dynamic changes of data and the diversity of retrieval requirements, and provide a high-quality data foundation for subsequent analysis and application.

[0131] The above-mentioned data mining device based on multimodal collaboration includes a processor and a memory, and the above-mentioned first retrieval unit, termination judgment unit and multimodal retrieval unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0132] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem in the prior art that the search accuracy is reduced due to the search request based on single modal information without related data search.

[0133] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0134] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the data mining method based on multimodal collaboration.

[0135] Specifically, the data mining methods based on multimodal collaboration include:

[0136] Step S201, a search step, receiving a search data request, and performing a search according to the search data in the search data request to obtain a current search result, and storing the current search result in a search request result set, wherein the search data request at least includes the time when the client initiates the request and the search data, and the search request result set includes multiple search results;

[0137] Step S202, a termination judgment step, performing a termination condition judgment based on the above-mentioned search request result set, wherein the above-mentioned termination condition includes that the total number of the above-mentioned search results in the above-mentioned search request result set meets the set total number of searches, or the number of search iterations reaches the set maximum number of iterations, and the above-mentioned number of search iterations is the total number of searches performed according to the above-mentioned search data request;

[0138] Step S203, a multimodal search step, in which, when the above search request result set does not satisfy the above termination condition, a multimodal search is performed to obtain a multimodal search result;

[0139] Step S204, repeating steps, mining related data according to the above multimodal retrieval results and the above retrieval request result set, and receiving a new retrieval data request initiated according to the above related data, repeating the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repeating step in sequence at least once until the above retrieval request result set meets the above termination condition, and outputting the above retrieval request result set to the above client.

[0140] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data mining method based on multimodal collaboration is executed when the program is running.

[0141] Specifically, the data mining methods based on multimodal collaboration include:

[0142] Step S201, a search step, receiving a search data request, and performing a search according to the search data in the search data request to obtain a current search result, and storing the current search result in a search request result set, wherein the search data request at least includes the time when the client initiates the request and the search data, and the search request result set includes multiple search results;

[0143] Step S202, a termination judgment step, performing a termination condition judgment based on the above-mentioned search request result set, wherein the above-mentioned termination condition includes that the total number of the above-mentioned search results in the above-mentioned search request result set meets the set total number of searches, or the number of search iterations reaches the set maximum number of iterations, and the above-mentioned number of search iterations is the total number of searches performed according to the above-mentioned search data request;

[0144] Step S203, a multimodal search step, in which, when the above search request result set does not satisfy the above termination condition, a multimodal search is performed to obtain a multimodal search result;

[0145] Step S204, repeating steps, mining related data according to the above multimodal retrieval results and the above retrieval request result set, and receiving a new retrieval data request initiated according to the above related data, repeating the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repeating step in sequence at least once until the above retrieval request result set meets the above termination condition, and outputting the above retrieval request result set to the above client.

[0146] An embodiment of the present invention provides a data mining and retrieval system. The data mining and retrieval system includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:

[0147] Step S201, a search step, receiving a search data request, and performing a search according to the search data in the search data request to obtain a current search result, and storing the current search result in a search request result set, wherein the search data request at least includes the time when the client initiates the request and the search data, and the search request result set includes multiple search results;

[0148] Step S202, a termination judgment step, performing a termination condition judgment based on the above-mentioned search request result set, wherein the above-mentioned termination condition includes that the total number of the above-mentioned search results in the above-mentioned search request result set meets the set total number of searches, or the number of search iterations reaches the set maximum number of iterations, and the above-mentioned number of search iterations is the total number of searches performed according to the above-mentioned search data request;

[0149] Step S203, a multimodal search step, in which, when the above search request result set does not satisfy the above termination condition, a multimodal search is performed to obtain a multimodal search result;

[0150] Step S204, repeating steps, mining related data according to the above multimodal retrieval results and the above retrieval request result set, and receiving a new retrieval data request initiated according to the above related data, repeating the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repeating step in sequence at least once until the above retrieval request result set meets the above termination condition, and outputting the above retrieval request result set to the above client.

[0151] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0152] Step S201, a search step, receiving a search data request, and performing a search according to the search data in the search data request to obtain a current search result, and storing the current search result in a search request result set, wherein the search data request at least includes the time when the client initiates the request and the search data, and the search request result set includes multiple search results;

[0153] Step S202, a termination judgment step, performing a termination condition judgment based on the above-mentioned search request result set, wherein the above-mentioned termination condition includes that the total number of the above-mentioned search results in the above-mentioned search request result set meets the set total number of searches, or the number of search iterations reaches the set maximum number of iterations, and the above-mentioned number of search iterations is the total number of searches performed according to the above-mentioned search data request;

[0154] Step S203, a multimodal search step, in which, when the above search request result set does not satisfy the above termination condition, a multimodal search is performed to obtain a multimodal search result;

[0155] Step S204, repeating steps, mining related data according to the above multimodal retrieval results and the above retrieval request result set, and receiving a new retrieval data request initiated according to the above related data, repeating the above retrieval step, the above termination judgment step, the above multimodal retrieval step and the repeating step in sequence at least once until the above retrieval request result set meets the above termination condition, and outputting the above retrieval request result set to the above client.

[0156] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0158] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0163] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0165] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0166] 1) The data mining method based on multimodal collaboration of the present application includes, first, a retrieval step, receiving a retrieval data request, and performing a search based on the retrieval data in the retrieval data request to obtain a current retrieval result, and storing the current retrieval result in a retrieval request result set, wherein the retrieval data request includes at least the time when the client initiated the request and the retrieval data, and the retrieval request result set includes multiple retrieval results; then, a termination judgment step, performing a termination condition judgment based on the retrieval request result set, wherein the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the number of retrieval iterations reaches the set maximum number of retrievals. The number of iterations, the above-mentioned number of retrieval iterations is the total number of retrievals performed according to the above-mentioned retrieval data request; thereafter, in the multimodal retrieval step, when the above-mentioned retrieval request result set does not meet the above-mentioned termination condition, a multimodal retrieval is performed to obtain a multimodal retrieval result; finally, a repeating step is performed to mine related data according to the above-mentioned multimodal retrieval results and the above-mentioned retrieval request result set, and a new retrieval data request initiated according to the above-mentioned related data is received, and the above-mentioned retrieval step, the above-mentioned termination judgment step, the above-mentioned multimodal retrieval step and the repeating step are sequentially repeated at least once until the above-mentioned retrieval request result set meets the above-mentioned termination condition, and the above-mentioned retrieval request result set is output to the above-mentioned client. The present application first obtains retrieval data according to the retrieval data request for retrieval; then a termination judgment is performed according to all retrieval results (i.e., the above-mentioned retrieval request result set), and if the termination condition is met, the retrieval is terminated and the retrieval request result is output; if the termination condition is not met, a multimodal retrieval is performed; then related data mining is performed according to the multimodal retrieval results and the retrieval request result set, and finally the related data obtained by the related data mining is used as a new retrieval request for re-retrieval. The selection of related data fully utilizes the cross-modal understanding and generation capabilities of the multimodal large model, overcomes the problem of insufficient expression of single modal information, and combines with the proposed feature enhancement, collaborative feature construction and multimodal collaborative similarity measurement network to accurately select related data as new retrieval requests; termination judgment is an adaptive termination judgment method, which automatically updates the termination parameters according to the retrieval results, so as to achieve the purpose of efficiently and accurately retrieving data that meets the requirements from massive data. This application solves the problem in the prior art that the retrieval accuracy is reduced by relying on single modal information for retrieval according to the retrieval request without performing related data retrieval.

[0167] 2) The data mining device based on multimodal collaboration of the present application comprises a first retrieval unit, which is used to execute a retrieval step, receive a retrieval data request, and perform a search according to the retrieval data in the retrieval data request to obtain a current retrieval result, and store the current retrieval result in a retrieval request result set, wherein the retrieval data request at least includes the initiation time of the client request and the retrieval data, and the retrieval request result set includes multiple retrieval results; a termination judgment unit, which is used to execute a termination judgment step, and perform a termination condition judgment according to the retrieval request result set, wherein the termination condition includes that the total number of the retrieval results in the retrieval request result set meets the set total number of retrievals, or the number of retrieval iterations reaches the set maximum number of iterations. The number of generations, the number of retrieval iterations is the total number of retrievals performed according to the retrieval data request; a multimodal retrieval unit, used to execute the multimodal retrieval step, and to perform a multimodal retrieval to obtain a multimodal retrieval result when the retrieval request result set does not meet the termination condition; a first repetition unit, used to execute the repetition step, to mine related data according to the multimodal retrieval result and the retrieval request result set, and to receive a new retrieval data request initiated according to the related data, and to repeat the retrieval step, the termination judgment step, the multimodal retrieval step and the repetition step at least once in sequence until the retrieval request result set meets the termination condition, and output the retrieval request result set to the client. The present application first obtains retrieval data according to the retrieval data request for retrieval; then performs a termination judgment according to all retrieval results (i.e., the retrieval request result set); if the termination condition is met, the retrieval is terminated and the retrieval request result is output; if the termination condition is not met, a multimodal retrieval is performed; then, related data mining is performed according to the multimodal retrieval result and the retrieval request result set, and finally, the related data obtained by the related data mining is used as a new retrieval request for re-retrieval. The selection of related data fully utilizes the cross-modal understanding and generation capabilities of the multimodal large model, overcomes the problem of insufficient expression of single modal information, and combines with the proposed feature enhancement, collaborative feature construction and multimodal collaborative similarity measurement network to accurately select related data as new retrieval requests; termination judgment is an adaptive termination judgment method, which automatically updates the termination parameters according to the retrieval results, so as to achieve the purpose of efficiently and accurately retrieving data that meets the requirements from massive data. This application solves the problem in the prior art that the retrieval accuracy is reduced by relying on single modal information for retrieval according to the retrieval request without performing related data retrieval.

[0168] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data mining method based on multimodal collaboration, characterized in that: include: A retrieval step, receiving a retrieval data request, and performing a search according to the retrieval data in the retrieval data request to obtain a current retrieval result, and storing the current retrieval result in a retrieval request result set, wherein the retrieval data request at least includes a request initiation time of the client and the retrieval data, and the retrieval request result set includes a plurality of retrieval results; a termination judgment step, performing a termination condition judgment based on the search request result set, wherein the termination condition includes that the total number of search results in the search request result set satisfies a set total number of searches, or the number of search iterations reaches a set maximum number of iterations, wherein the number of search iterations is the total number of searches performed based on the search data request; A multimodal search step, in which, when the search request result set does not satisfy the termination condition, a multimodal search is performed to obtain a multimodal search result; The repeating step mines related data according to the multimodal retrieval result and the retrieval request result set, and receives a new retrieval data request initiated according to the related data, and sequentially repeats the retrieval step, the termination judgment step, the multimodal retrieval step and the repeating step at least once until the retrieval request result set meets the termination condition, and outputs the retrieval request result set to the client.

2. The method according to claim 1, characterized in that The search data is one of the following: first search data and second search data, the first search data is a target search keyword sent by the client when performing a search for the first time, and the second search data is a search result in the associated data that is not repeated with the search request result set. After receiving the search data request, the method further includes: Determining whether the first retrieval data exists in the retrieval data request; In the case that the first search data does not exist in the search data request, a first prompt message is issued and the search is terminated, wherein the first prompt message is used to prompt that the search data request issued by the client does not contain valid search data and cannot be searched; When the first search data exists in the search data request, the search is performed according to the first search data in the search data request.

3. The method according to claim 1, characterized in that: The termination condition is judged according to the search request result set, including: Determine whether the total number of the search results in the search request result set is greater than or equal to the set total number of searches; When the total number is greater than or equal to the set total number of searches, determining that the search request result set meets the termination condition, and terminating the search; When the amount of search result data is less than the set total number of searches, determining whether the number of search iterations reaches the set maximum number of iterations; When the number of search iterations reaches the set maximum number of iterations, determining that the search request result set meets the termination condition, and terminating the search; When the number of retrieval iterations does not reach the set maximum number of iterations, a multimodal retrieval is performed to obtain a multimodal retrieval result.

4. The method according to claim 1, characterized in that: After performing the multimodal search and obtaining the multimodal search results, the method further includes: Calculating the similarity between each of the multimodal search results and the search data request to obtain a plurality of multimodal search similarities; Calculating an average value based on all the multimodal retrieval similarities to obtain an average similarity value; When the similarity average value is greater than the set evaluation threshold, increasing the set maximum number of iterations; When the similarity average value is less than or equal to the set evaluation threshold, the set maximum number of iterations is reduced.

5. The method according to claim 1, characterized in that The retrieval request result set also includes the similarity corresponding to each of the retrieval results, where the similarity is the degree of similarity between the retrieval result and the retrieval data request. Before mining related data based on the multimodal retrieval result and the retrieval request result set, the method further includes: If there is a latest search result in the multimodal search result, the latest search result is added to the search request result set, and the latest search result is a search result in the multimodal search result that is not repeated in the search request result set; In the event that the multimodal retrieval result is repeated with the retrieval result stored in the retrieval request result set, a weighted addition is performed on the similarity of the multimodal retrieval result and the similarity of the repeated retrieval result, and the similarity corresponding to the repeated retrieval result is updated. The repeated retrieval result is the retrieval result in the retrieval request result set that is repeated with the multimodal retrieval result.

6. The method according to claim 1, characterized in that The retrieval request result set also includes the similarity corresponding to each of the retrieval results, where the similarity is the degree of similarity between the retrieval result and the retrieval data request, and mining related data according to the multimodal retrieval result and the retrieval request result set includes: Merge the multimodal search result and the search request result set to remove duplicates, and update the search request result set, wherein the deduplication process is used to remove duplicate search results; Selecting a first set number of search results in the search request result set in descending order of the similarities to obtain associated search results; Performing feature extraction, clustering processing and feature enhancement processing on the associated search results in sequence to obtain enhanced features, and simultaneously performing semantic enhancement processing on the associated search results using a visual language algorithm to obtain semantic features; A collaborative feature is constructed based on the enhanced feature and the semantic feature in combination with a first formula, where the first formula is: f1 represents the enhanced feature, f2 represents the semantic feature, F s represents the synergistic feature, Indicates the similarity between the enhanced feature and the semantic feature; Input the collaborative feature and the first search data into a similarity measurement network, and output a ranking result, wherein the first search data is a target search keyword sent by the client when performing the first search, and the ranking result is a result obtained by sorting the similarity between the collaborative feature and the first search data from large to small; A second set number of the collaborative features are selected according to the sorting result to obtain the associated data.

7. The method according to claim 1, characterized in that After receiving a new request for retrieving data based on the associated data, the method further includes: Determine whether there is second search data in the associated data, where the second search data is a search result in the associated data that is not repeated with the search request result set; In the case that the second search result does not exist in the associated data, issuing a second prompt message and terminating the search, wherein the second prompt message is used to prompt that the second search data does not exist in the associated data; In the case that the second search result exists in the associated data, the search step, the termination judgment step, the multimodal search step and the repeating step are repeated at least once until the search request result set meets the termination condition.

8. A data mining device based on multimodal collaboration, characterized in that: The device comprises: A first retrieval unit is used to perform a retrieval step, receive a retrieval data request, perform retrieval according to the retrieval data in the retrieval data request, obtain a current retrieval result, and store the current retrieval result in a retrieval request result set, wherein the retrieval data request at least includes a request initiation time of the client and the retrieval data, and the retrieval request result set includes multiple retrieval results; a termination judgment unit, configured to execute a termination judgment step, and to judge a termination condition according to the retrieval request result set, wherein the termination condition includes that the total number of retrieval results in the retrieval request result set satisfies a set total number of retrievals, or the number of retrieval iterations reaches a set maximum number of iterations, wherein the number of retrieval iterations is the total number of retrievals performed according to the retrieval data request; A multimodal retrieval unit, configured to execute a multimodal retrieval step, and to perform a multimodal retrieval to obtain a multimodal retrieval result when the retrieval request result set does not satisfy the termination condition; The first repeating unit is used to execute the repeating step, mine related data according to the multimodal retrieval result and the retrieval request result set, receive a new retrieval data request initiated according to the related data, and repeat the retrieval step, the termination judgment step, the multimodal retrieval step and the repeating step in sequence at least once until the retrieval request result set meets the termination condition, and output the retrieval request result set to the client.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images