An efficient image-text retrieval method and system based on multimodal feature fusion

By extracting the feature information of the graphic and text search data, judging and identifying parameters, and optimizing resource allocation, the problems of high computational complexity and inefficiency in the graphic and text search method of multimodal feature fusion graphic and text search method are solved, and efficient and accurate graphic and text are achieved. Search and resource management.

CN118349699BActive Publication Date: 2024-11-15WUHAN QIUYE EDUCATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410578440.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-11
Publication Date
2024-11-15
Estimated Expiration
2044-05-11

AI Technical Summary

Technical Problem

The existing multimodal feature fusion graphic search method has high computational complexity, poor feature fusion effect, and inability to effectively utilize system resources, resulting in low retrieval efficiency and accuracy, and waste and insufficient resources.

Method used

By obtaining graphic and text search data, extracting feature information, calculating identification parameters, determining whether it meets the preset conditions, marking it as fuzzy feature information, calculating system resource utilization data, optimizing resource allocation, and adjusting resource allocation strategies to improve retrieval efficiency and accuracy.

Benefits of technology

It improves the accuracy and efficiency of graphic and text retrieval, ensures the effective utilization of system resources, adapts to changes in system resource utilization, and improves the stability and adaptability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118349699B_ABST
    Figure CN118349699B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of multimodal retrieval technology, and specifically relates to an efficient image and text retrieval method and system based on multimodal feature fusion. The invention can improve the accuracy and efficiency of retrieval by fusing these feature information, especially for data containing image and text information, by presetting conditions to judge the image and text retrieval recognition parameters, it can filter out data that is not suitable for rapid retrieval to a certain extent, improve retrieval efficiency, optimize the allocation of retrieval tasks according to the actual utilization of system resources, ensure the effective utilization of system resources, thereby improving overall performance, and use rapid retrieval of system resource utilization fluctuation data to more flexibly adjust resource allocation, adapt to changes in system resource utilization, and improve stability and adaptability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal retrieval, and in particular relates to an efficient image-text retrieval method and system based on multimodal feature fusion. Background Art

[0002] With the rapid development of information technology and the widespread application of technologies such as big data, cloud computing and artificial intelligence, we are faced with more and more multimedia information, including text, images, audio and video. These multimedia information often exist in different modal forms. How to effectively process and understand these multimodal information has become an important research issue. Especially in the field of image and text retrieval, how to effectively integrate the two modal information of text and image to achieve more accurate retrieval results is a hot topic and difficulty in research.

[0003] Existing image and text retrieval methods can be mainly divided into two categories: retrieval methods based on a single modality and retrieval methods based on multimodal fusion. Retrieval methods based on a single modality, whether text-based or image-based, ignore the information of the other modality, thereby limiting the accuracy and efficiency of the retrieval. Although retrieval methods based on multimodal fusion have solved this problem to a certain extent, existing multimodal fusion methods often have problems such as high computational complexity and unsatisfactory feature fusion effects, making it difficult to meet the needs of practical applications.

[0004] However, the existing efficient image and text retrieval with multimodal feature fusion cannot select the appropriate system resource calling algorithm according to the clarity of the input text and image data, which reduces the user's retrieval efficiency and accuracy, reduces the rational use of system resources, and causes waste and shortage of system resources. Summary of the invention

[0005] The purpose of the present invention is to provide an efficient image and text retrieval method based on multimodal feature fusion, which can improve the efficiency and accuracy of image and text retrieval by effectively utilizing multimodal features, preset condition judgment and resource utilization optimization, so that the system can better meet the needs in actual application scenarios.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An efficient image-text retrieval method based on multimodal feature fusion, comprising:

[0008] Obtaining input image-text retrieval data, and extracting image-text retrieval feature information from the image-text retrieval data;

[0009] Calculate the image and text retrieval recognition parameters according to the image and text retrieval feature information, and determine whether the image and text retrieval recognition parameters meet the preset conditions;

[0010] If the image-text retrieval recognition parameters do not meet the preset conditions, the image-text retrieval feature information is determined to not meet the quick retrieval conditions and is marked as fuzzy image-text feature information;

[0011] Calculate the plan to retrieve system resource utilization data based on fuzzy graphic feature information;

[0012] Get quick retrieval of system resource utilization fluctuation data;

[0013] Obtaining available retrieval system resource utilization data according to rapid retrieval system resource utilization fluctuation data;

[0014] Calculate actual retrieval system resource utilization data based on available retrieval system resource utilization data and planned retrieval system resource utilization data;

[0015] The retrieval time is obtained based on the actual retrieval system resource utilization data.

[0016] In a preferred solution, the step of obtaining input image-text retrieval data and extracting image-text retrieval feature information from the image-text retrieval data includes:

[0017] Get the input image and text retrieval data;

[0018] Extracting text retrieval feature information and image retrieval feature information from image-text retrieval data;

[0019] The text retrieval feature information and the image retrieval feature information are aggregated into the image-text retrieval feature information.

[0020] In a preferred solution, the step of calculating the image-text retrieval identification parameter according to the image-text retrieval feature information and judging whether the image-text retrieval identification parameter meets the preset conditions includes:

[0021] Obtain corresponding text retrieval feature parameters and image retrieval feature parameters according to the image-text retrieval feature information;

[0022] Obtaining text standard retrieval feature parameters and image standard retrieval feature parameters;

[0023] Get the image and text retrieval recognition function;

[0024] Inputting text retrieval feature parameters, image retrieval feature parameters, text standard retrieval feature parameters and image standard retrieval feature parameters into a graphic-text retrieval recognition function, and marking the output result as a graphic-text retrieval recognition parameter;

[0025] Obtaining a standard image and text search and recognition evaluation interval, and determining whether the image and text search and recognition parameters are within the standard image and text search and recognition evaluation interval;

[0026] If the image-text retrieval recognition parameter is within the image-text standard retrieval recognition evaluation interval, it is determined that the image-text retrieval feature information meets the fast retrieval condition;

[0027] If the image-text retrieval recognition parameter is not within the image-text standard retrieval recognition evaluation interval, it is determined that the image-text retrieval feature information does not meet the quick retrieval conditions.

[0028] In a preferred embodiment, the step of calculating a plan to retrieve system resource utilization data based on fuzzy graphic feature information includes:

[0029] Obtain corresponding image-text retrieval recognition parameters according to fuzzy image-text feature information;

[0030] Obtaining standard image and text retrieval identification parameters and image and text standard retrieval system resource utilization;

[0031] Get the plan to retrieve the system resource utilization function;

[0032] The image-text retrieval identification parameters, image-text standard retrieval identification parameters and image-text standard retrieval system resource utilization are input into the planned retrieval system resource utilization function, and the output result is marked as the planned retrieval system resource utilization to obtain the planned retrieval system resource utilization data.

[0033] In a preferred solution, the step of obtaining rapid retrieval system resource utilization fluctuation data includes:

[0034] Obtaining the image and text retrieval feature information corresponding to the retrieval identification parameters that meet the preset conditions, and marking it as clear image and text retrieval feature information;

[0035] Obtaining the system resource utilization rate of the clear image and text retrieval feature information when performing image and text retrieval, and marking it as the rapid retrieval system resource utilization rate, wherein the rapid retrieval system resource utilization rate includes the historical rapid retrieval system resource utilization rate and the current rapid retrieval system resource utilization rate;

[0036] Get quick retrieval system resource utilization fluctuation function;

[0037] The historical rapid retrieval system resource utilization and the current rapid retrieval system resource utilization are input into the rapid retrieval system resource utilization fluctuation function, the output result is marked as the rapid retrieval system resource utilization fluctuation parameter, and the rapid retrieval system resource utilization fluctuation data is obtained.

[0038] In a preferred solution, the step of obtaining available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data comprises:

[0039] Get the current system resource utilization;

[0040] Obtaining a rapid retrieval standard system resource utilization fluctuation parameter and a rapid retrieval standard system resource utilization rate;

[0041] Get available retrieval system resource utilization function;

[0042] The current system resource utilization, the quick retrieval system resource utilization fluctuation parameter, the quick retrieval standard system resource utilization fluctuation parameter and the quick retrieval standard system resource utilization are input into the available retrieval system resource utilization function, and the output result is marked as the available retrieval system resource utilization to obtain the available retrieval system resource utilization data.

[0043] In a preferred solution, the step of calculating the actual retrieval system resource utilization data based on the available retrieval system resource utilization data and the planned retrieval system resource utilization data comprises:

[0044] Obtaining the corresponding available retrieval system resource utilization rate according to the available retrieval system resource utilization data, and obtaining the corresponding planned retrieval system resource utilization rate according to the planned retrieval system resource utilization data;

[0045] Get the actual retrieval volatility function;

[0046] Input the available retrieval system resource utilization and the planned retrieval system resource utilization into the actual retrieval fluctuation function, and mark the output result as the actual retrieval fluctuation value;

[0047] Determine whether the actual retrieval fluctuation value is greater than zero;

[0048] If the actual search fluctuation value is less than or equal to zero, then the available search system resource utilization is determined to be the actual search system resource utilization data;

[0049] If the actual retrieval fluctuation value is greater than zero, it is determined that the planned retrieval system resource utilization is the actual retrieval system resource utilization data.

[0050] In a preferred solution, the step of obtaining the search duration according to the actual search system resource utilization data includes:

[0051] Acquire a search schedule, wherein the search schedule includes a plurality of search system resource utilization data and a search duration corresponding to each search system resource utilization data;

[0052] The corresponding search duration is obtained in the search timetable according to the actual search system resource utilization data.

[0053] The present invention also provides an efficient image-text retrieval system based on multimodal feature fusion, which is used for the above-mentioned efficient image-text retrieval method based on multimodal feature fusion, comprising:

[0054] The image-text retrieval module is used to obtain input image-text retrieval data and extract image-text retrieval feature information from the image-text retrieval data;

[0055] A search judgment module is used to calculate the image and text search recognition parameters according to the image and text search feature information, and to judge whether the image and text search recognition parameters meet the preset conditions;

[0056] If the image-text retrieval recognition parameters do not meet the preset conditions, the image-text retrieval feature information is determined to not meet the quick retrieval conditions and is marked as fuzzy image-text feature information;

[0057] A planning resource module, used to calculate the planning retrieval system resource utilization data according to the fuzzy graphic feature information;

[0058] Fluctuation module, used to obtain and quickly retrieve system resource utilization fluctuation data;

[0059] An available resource module, used to obtain available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data;

[0060] An actual resource module, used for calculating actual retrieval system resource utilization data according to available retrieval system resource utilization data and planned retrieval system resource utilization data;

[0061] The duration module is used to obtain the retrieval duration based on the actual retrieval system resource utilization data.

[0062] And, an efficient image and text retrieval terminal based on multimodal feature fusion, comprising:

[0063] one or more processors;

[0064] a storage device having one or more programs stored thereon;

[0065] When the one or more programs are executed by the one or more processors, the one or more processors implement the efficient image-text retrieval method based on multimodal feature fusion.

[0066] The technical effects achieved by the present invention are:

[0067] The present invention can improve the accuracy and efficiency of retrieval by fusing these feature information, especially for data containing image and text information. By presetting conditions to judge the image and text retrieval recognition parameters, data that is not suitable for rapid retrieval can be filtered out to a certain extent, thereby improving the retrieval efficiency. According to the actual utilization of system resources, the retrieval tasks are optimized and allocated to ensure the effective utilization of system resources, thereby improving the overall performance. By using the rapid retrieval of system resource utilization fluctuation data, resource allocation can be adjusted more flexibly to adapt to changes in system resource utilization and improve stability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of the method provided by the present invention;

[0069] Figure 2 It is a system module diagram provided by the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0071] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0072] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0073] Secondly, the present invention is described in detail in conjunction with schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the schematic diagrams are only examples and should not limit the scope of protection of the present invention.

[0074] Please see attached Figure 1 As shown, an efficient image-text retrieval method based on multimodal feature fusion is provided, including:

[0075] S1, obtaining input image-text retrieval data, and extracting image-text retrieval feature information from the image-text retrieval data;

[0076] S2, calculating the image and text retrieval recognition parameters according to the image and text retrieval feature information, and determining whether the image and text retrieval recognition parameters meet the preset conditions;

[0077] If the image-text retrieval recognition parameters do not meet the preset conditions, the image-text retrieval feature information is determined to not meet the quick retrieval conditions and is marked as fuzzy image-text feature information;

[0078] S3, calculating a plan to retrieve system resource utilization data based on the fuzzy graphic feature information;

[0079] S4, obtaining rapid retrieval system resource utilization fluctuation data;

[0080] S5, obtaining available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data;

[0081] S6, calculating actual retrieval system resource utilization data according to the available retrieval system resource utilization data and the planned retrieval system resource utilization data;

[0082] S7. Obtain the search duration based on the actual search system resource utilization data.

[0083] As in the above steps S1 to S7, first, the image and text data input by the user is obtained, and the retrieval feature information of the image and text is extracted therefrom. The feature information may include the color histogram, texture features, keywords of the text, etc. of the image, and the image and text retrieval recognition parameters are calculated using the extracted image and text retrieval feature information, and compared with the preset conditions. If the recognition parameters meet the preset conditions, a quick retrieval is directly performed according to the preset system resource utilization rate. Otherwise, the feature information is marked as fuzzy image and text feature information, indicating that it is not suitable for quick retrieval. For the image and text feature information marked as fuzzy, the corresponding retrieval system resource utilization data is calculated, which can help evaluate its required in the actual retrieval process. Resource consumption, collects and records the resource utilization fluctuation data of the fast retrieval system over a period of time, which helps to understand the changes in system resource utilization. According to the resource utilization fluctuation data of the fast retrieval system, the currently available retrieval system resource utilization data, that is, the resources that can be allocated to the actual retrieval task, is determined. The system resource utilization required to actually perform the retrieval task is calculated by combining the available retrieval system resource utilization data and the planned retrieval system resource utilization data. Finally, based on the calculated actual retrieval system resource utilization data, the time required to perform the retrieval task is estimated. Suppose we have an online shopping platform where users can search for products they are interested in by entering text descriptions or uploading pictures. Products, this platform hopes to improve the efficiency and accuracy of search to enhance user experience and promote sales. Users may enter text descriptions, such as "red dress", and may upload a picture showing a red dress. The corresponding feature information, such as color, shape, texture, etc., is extracted based on the input text description and picture. Then, the image and text retrieval recognition parameters are calculated based on these feature information. If it is judged that these parameters can accurately match the user's needs, a quick search can be performed directly to save system resources. However, if the match is considered to be low, the request may be marked as fuzzy image and text feature information, and then the search is planned according to resource utilization. While ensuring the accuracy of the search, Reasonable use of system resources improves overall efficiency and provides users with a better search experience. It utilizes the multimodal features of images and texts. By integrating these feature information, the accuracy and efficiency of retrieval can be improved, especially for data containing image and text information. By pre-setting conditions to judge the image and text retrieval recognition parameters, it can filter out data that is not suitable for fast retrieval to a certain extent, improve retrieval efficiency, and optimize the allocation of retrieval tasks according to the actual utilization of system resources to ensure the effective utilization of system resources, thereby improving overall performance. By using fast retrieval of system resource utilization fluctuation data, resource allocation can be adjusted more flexibly to adapt to changes in system resource utilization and improve stability and adaptability.

[0084] The step of obtaining input image-text retrieval data and extracting image-text retrieval feature information from the image-text retrieval data includes:

[0085] S101, obtaining input image and text retrieval data;

[0086] S102, extracting text retrieval feature information and image retrieval feature information from the image-text retrieval data;

[0087] S103: Aggregate the text retrieval feature information and the image retrieval feature information into image-text retrieval feature information.

[0088] As in the above steps S101 to S103, firstly, the image and text retrieval data input by the user is obtained from an external source. Such data may be images and texts uploaded by the user, or data obtained from the network, database or other channels. The obtained image and text retrieval data is processed to extract feature information from the text and image aspects respectively. For text data, feature information such as word frequency, TF-IDF value, word vector, topic model, etc. may be extracted to represent the key features of the text content. For image data, feature information such as color histogram, texture feature, shape feature, feature vector extracted by deep learning model, etc. may be extracted to describe the visual features of the image. Finally, the image and text retrieval data are processed to extract feature information such as the image and text. The text retrieval feature information and image retrieval feature information extracted from the data are summarized to form complete image-text retrieval feature information. Such feature information can be used for subsequent image-text retrieval tasks. By extracting feature information of both text and image, the diversity of image-text retrieval data can be fully utilized, thereby improving the comprehensiveness and accuracy of the retrieval. The summary of text and image feature information can provide a richer and more comprehensive feature description, enabling a better understanding and representation of image and text content, thereby improving the retrieval effect. The extracted image-text retrieval feature information can be used for subsequent retrieval tasks, such as similar image retrieval, related text retrieval, etc., thereby providing users with more accurate and rich retrieval results.

[0089] The step of calculating the image-text retrieval identification parameter according to the image-text retrieval feature information and judging whether the image-text retrieval identification parameter meets the preset conditions includes:

[0090] S201, acquiring corresponding text retrieval feature parameters and image retrieval feature parameters according to the image-text retrieval feature information;

[0091] S202, obtaining text standard retrieval feature parameters and image standard retrieval feature parameters;

[0092] S203, obtaining a picture and text search recognition function;

[0093] S204, inputting the text retrieval feature parameters, the image retrieval feature parameters, the text standard retrieval feature parameters and the image standard retrieval feature parameters into the image-text retrieval recognition function, and outputting the result marked as the image-text retrieval recognition parameter;

[0094] S205, obtaining a standard image and text search and recognition evaluation interval, and determining whether the image and text search and recognition parameters are within the standard image and text search and recognition evaluation interval;

[0095] If the image-text retrieval recognition parameter is within the image-text standard retrieval recognition evaluation interval, it is determined that the image-text retrieval feature information meets the fast retrieval condition;

[0096] If the image-text retrieval recognition parameter is not within the image-text standard retrieval recognition evaluation interval, it is determined that the image-text retrieval feature information does not meet the quick retrieval conditions.

[0097] As in the above steps S201 to S205, the corresponding text retrieval feature parameters and image retrieval feature parameters are extracted based on the input image and text retrieval feature information. These parameters are numerical representations of text and image features. Standard retrieval feature parameters for text and images are predefined and used as reference standards to determine whether the image and text retrieval feature information meets the preset conditions. According to actual needs and task scenarios, an appropriate image and text retrieval recognition function is designed or selected. The function can be a mathematical model, a machine learning model or a set of rules. The image and text retrieval recognition function is , where S represents the image and text retrieval recognition parameter, W represents the text retrieval feature parameter, and T represents the text retrieval feature parameter. Represents the image standard retrieval feature parameters, It is represented as text standard retrieval feature parameters, and the extracted text retrieval feature parameters, image retrieval feature parameters, and preset text standard retrieval feature parameters and image standard retrieval feature parameters are used as input. The image-text retrieval recognition parameters are calculated by the image-text retrieval recognition function. The image-text standard retrieval recognition evaluation interval is pre-set, which represents the range that meets the rapid retrieval conditions. According to this interval, it is judged whether the calculated image-text retrieval recognition parameters are within it. If they are within the evaluation interval, the image-text retrieval feature information is judged to meet the rapid retrieval conditions; otherwise, it is judged not to meet the rapid retrieval conditions. By defining the standard retrieval feature parameters of text and images and the image-text standard retrieval recognition evaluation interval, it is possible to quantitatively judge whether the image-text retrieval feature information meets the preset conditions, thereby avoiding the uncertainty of subjective judgment. By judging whether the image-text retrieval recognition parameters are within the preset evaluation interval, the feature information that does not meet the rapid retrieval conditions can be quickly filtered out, thereby improving the retrieval efficiency. The set evaluation interval can be adjusted according to the actual situation to adapt to different application scenarios and needs, thereby ensuring flexibility and adaptability.

[0098] The step of calculating a plan to retrieve system resource utilization data according to the fuzzy graphic feature information comprises:

[0099] S301, obtaining corresponding image and text retrieval recognition parameters according to fuzzy image and text feature information;

[0100] S302, obtaining standard image and text search identification parameters and standard image and text search system resource utilization;

[0101] S303, obtaining a plan retrieval system resource utilization function;

[0102] S304, inputting the image and text retrieval identification parameters, the image and text standard retrieval identification parameters and the image and text standard retrieval system resource utilization into the planned retrieval system resource utilization function, and marking the output result as the planned retrieval system resource utilization to obtain the planned retrieval system resource utilization data.

[0103] As in the above steps S301 to S304, the corresponding image and text retrieval recognition parameters are obtained according to the fuzzy image and text feature information to describe the retrieval capability or other correlation indicators of the fuzzy feature information. The standard retrieval recognition parameters of the image and text, as well as the standard retrieval system resource utilization are predefined. These parameters and rate values ​​can be set according to actual needs and system configuration. According to actual needs and task scenarios, an appropriate planned retrieval system resource utilization function is designed or selected. The function can be a mathematical model, a machine learning model or a set of rules, which is used to calculate the planned retrieval system resource utilization according to the input parameters. The planned retrieval system resource utilization function is In the formula, J represents the resource utilization of the planned retrieval system, L represents the resource utilization of the standard image and text retrieval system, and B represents the standard image and text retrieval identification parameters. The acquired image and text retrieval identification parameters, standard retrieval identification parameters and standard retrieval system resource utilization are used as inputs. The resource utilization of the planned retrieval system is calculated by the resource utilization function of the planned retrieval system. This value can represent the resource utilization under the current fuzzy feature situation. The resource utilization data of the planned retrieval system is calculated based on the fuzzy image and text feature information. It can dynamically allocate resources according to the actual situation to meet the needs of the retrieval task and improve the utilization efficiency of system resources. Through the resource utilization function of the planned retrieval system, resources can be optimized and configured according to the image and text retrieval identification parameters and the standard retrieval parameters, making resource allocation more reasonable and avoiding resource waste or shortage. By pre-planning the resource utilization of the retrieval system, the resource utilization can be better grasped to ensure stable operation under high load conditions.

[0104] The step of obtaining the rapid retrieval system resource utilization fluctuation data comprises:

[0105] S401, obtaining image and text retrieval feature information corresponding to the retrieval identification parameters that meet the preset conditions, and marking it as clear image and text retrieval feature information;

[0106] S402, obtaining the system resource utilization rate of the clear image and text retrieval feature information when performing image and text retrieval, and marking it as the rapid retrieval system resource utilization rate, wherein the rapid retrieval system resource utilization rate includes the historical rapid retrieval system resource utilization rate and the current rapid retrieval system resource utilization rate;

[0107] S403, obtaining a rapid retrieval system resource utilization fluctuation function;

[0108] S404: input the historical rapid retrieval system resource utilization and the current rapid retrieval system resource utilization into the rapid retrieval system resource utilization fluctuation function, and mark the output result as the rapid retrieval system resource utilization fluctuation parameter to obtain the rapid retrieval system resource utilization fluctuation data.

[0109] As in the above steps S401 to S404, in the image and text retrieval system, the retrieval identification parameters that meet the requirements are screened out according to the preset conditions, that is, those parameters that can clearly identify the image and text features, and the corresponding image and text retrieval feature information is marked as clear image and text retrieval feature information. When performing the image and text retrieval task, the system resource utilization corresponding to the clear image and text retrieval feature information is recorded, including the historical rapid retrieval system resource utilization and the current rapid retrieval system resource utilization. These utilizations can be CPU usage, memory occupancy, network bandwidth utilization, etc., and an appropriate rapid retrieval system resource utilization fluctuation function is designed or selected to analyze and describe the fluctuation of system resource utilization. This function can be a statistical model, a time series model or other mathematical models. The rapid retrieval system resource utilization fluctuation function is , where i represents the number of the historical rapid retrieval system resource utilization, and n represents the total number of the historical rapid retrieval system resource utilization. It is represented as the i-th historical quick retrieval system resource utilization, and F is represented as the current quick retrieval system resource utilization. The obtained historical and current quick retrieval system resource utilization are taken as input, and the quick retrieval system resource utilization fluctuation parameter is calculated through the quick retrieval system resource utilization fluctuation function, that is, the data describing the change of resource utilization. By recording the system resource utilization corresponding to the clear graphic retrieval feature information and combining it with historical data, the utilization of system resources can be analyzed in real time, and the fluctuation of resource utilization can be discovered in time. With the help of the quick retrieval system resource utilization fluctuation function, the fluctuation trend of system resource utilization can be predicted, so as to adjust resource allocation in a targeted manner. By analyzing the quick retrieval system resource utilization fluctuation data, the resource allocation strategy can be optimized to make resource utilization more efficient, thereby improving the overall performance and user experience.

[0110] The step of obtaining available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data comprises:

[0111] S501, obtaining current system resource utilization;

[0112] S502, obtaining a rapid retrieval standard system resource utilization fluctuation parameter and a rapid retrieval standard system resource utilization rate;

[0113] S503, obtaining available retrieval system resource utilization function;

[0114] S504. Input the current system resource utilization, the quick retrieval system resource utilization fluctuation parameter, the quick retrieval standard system resource utilization fluctuation parameter and the quick retrieval standard system resource utilization into the available retrieval system resource utilization function, and mark the output result as the available retrieval system resource utilization to obtain the available retrieval system resource utilization data.

[0115] As in the above steps S501 to S504, first obtain the current system resource utilization, including CPU usage, memory occupancy, network bandwidth utilization, etc. These data can be obtained from the system monitoring tool or the interface provided by the operating system. Quick retrieval standard system resource utilization fluctuation parameters and standard system resource utilization are predefined. These parameters and rate values ​​can be set according to actual needs and system configuration. According to actual needs and task scenarios, design or select an appropriate available retrieval system resource utilization function, which can be a mathematical model, a machine learning model or a set of rules for calculating the available retrieval system resource utilization based on the input parameters. The available retrieval system resource utilization function is In the formula, H represents the available retrieval system resource utilization, D represents the current system resource utilization, C represents the fast retrieval standard system resource utilization, and V represents the fast retrieval standard system resource utilization fluctuation parameter. The current system resource utilization, the fast retrieval system resource utilization fluctuation parameter and the standard resource utilization are obtained as inputs, and the available retrieval system resource utilization is calculated by the available retrieval system resource utilization function. This value can represent the resource utilization that can be provided in the current environment. Through the calculated available retrieval system resource utilization, the resource allocation strategy can be dynamically adjusted to adapt to the fluctuation of the current system resource utilization, thereby improving the resource utilization efficiency. With the help of the available retrieval system resource utilization function, the resource allocation strategy can be optimized according to the current system resource utilization and the fast retrieval standard system resource utilization to ensure performance stability under high load conditions. By calculating the available retrieval system resource utilization according to the fast retrieval system resource utilization fluctuation data, the resource utilization can be better grasped to ensure normal operation under different load conditions.

[0116] The step of calculating the actual retrieval system resource utilization data according to the available retrieval system resource utilization data and the planned retrieval system resource utilization data comprises:

[0117] S601, obtaining the corresponding available retrieval system resource utilization rate according to the available retrieval system resource utilization data, and obtaining the corresponding planned retrieval system resource utilization rate according to the planned retrieval system resource utilization data;

[0118] S602, obtaining an actual search fluctuation function;

[0119] S603, inputting the available retrieval system resource utilization and the planned retrieval system resource utilization into the actual retrieval fluctuation function, and marking the output result as the actual retrieval fluctuation value;

[0120] S604, determining whether the actual search fluctuation value is greater than zero;

[0121] If the actual search fluctuation value is less than or equal to zero, then the available search system resource utilization is determined to be the actual search system resource utilization data;

[0122] If the actual retrieval fluctuation value is greater than zero, it is determined that the planned retrieval system resource utilization is the actual retrieval system resource utilization data.

[0123] As in the above steps S601 to S604, the available retrieval system resource utilization is obtained from the available retrieval system resource utilization data, and the planned retrieval system resource utilization is obtained from the planned retrieval system resource utilization data. These utilizations can be CPU usage, memory occupancy, network bandwidth utilization, etc., and an appropriate actual retrieval fluctuation function is designed or selected to analyze and describe the fluctuation of the actual system resource utilization. This function can be a statistical model, a time series model or other mathematical models. The actual retrieval fluctuation function is A=HJ, where A represents the actual retrieval fluctuation value. The obtained available retrieval system resource utilization and the planned retrieval system resource utilization are used as inputs, and the actual retrieval fluctuation value is calculated through the actual retrieval fluctuation function. This value can represent the actual system resource utilization in the current environment. It is determined whether the actual retrieval fluctuation value is greater than zero to determine the final actual retrieval system resource utilization data. If the actual retrieval fluctuation value is less than or equal to zero, it is determined. The available retrieval system resource utilization is the actual retrieval system resource utilization data, which means that the system resource utilization has large fluctuations and cannot be used stably. The available retrieval system resource utilization can be directly used as the actual utilization data. If the actual retrieval fluctuation value is greater than zero, the planned retrieval system resource utilization is determined to be the actual retrieval system resource utilization data, which means that the system resource utilization is within an acceptable fluctuation range. Therefore, the planned retrieval system resource utilization is used as the actual utilization data. By calculating the actual retrieval system resource utilization data, the resource allocation strategy can be dynamically adjusted according to the fluctuation of the current system resource utilization, so as to keep the system resource utilization within a reasonable range and improve the resource utilization efficiency. According to the judgment of the actual retrieval fluctuation value, it can be ensured that the resource utilization is within an acceptable fluctuation range. By judging the size of the actual retrieval fluctuation value, an optimization decision can be made to select the appropriate resource utilization as the actual retrieval system resource utilization data to ensure normal operation under different load conditions.

[0124] The step of obtaining the retrieval duration according to the actual retrieval system resource utilization data comprises:

[0125] S701, obtaining a search schedule, wherein the search schedule includes a plurality of search system resource utilization data and a search duration corresponding to each search system resource utilization data;

[0126] S702: Obtain corresponding search duration in the search time table according to actual search system resource utilization data.

[0127] As in the above steps S701 to S702, the retrieval schedule is a time schedule that records multiple retrieval tasks, including the retrieval system resource utilization data and the corresponding retrieval duration corresponding to each retrieval task. This table can be formulated according to actual conditions and is usually generated by a system administrator or a scheduling algorithm. The actual retrieval system resource utilization data obtained is compared with the data in the retrieval schedule to find the corresponding retrieval task, and the corresponding retrieval duration is extracted therefrom. This duration can be a time period in milliseconds, seconds, minutes or hours, which represents the time required for the retrieval task. The actual retrieval system resource utilization data reflects the time spent on executing a certain retrieval task. By comparing the actual utilization data and the retrieval schedule, the corresponding retrieval time under different resource utilization conditions can be found, revealing the relationship between resource utilization and time. Obtaining the retrieval time based on the actual retrieval system resource utilization data is helpful for the scheduling algorithm to optimize tasks and allocate resources. For example, task delays may occur under high resource utilization, and it is necessary to adjust the resource allocation or scheduling strategy to ensure that the retrieval time is not affected. By comparing the actual retrieval time with the expected time, the performance and efficiency of the retrieval system can be evaluated. If the actual retrieval time deviates significantly from expectations, it may be necessary to further optimize the system configuration or algorithm design to improve the response speed and stability.

[0128] Please see attached Figure 2 As shown, the present invention also provides an efficient image-text retrieval system based on multimodal feature fusion, which is used for the above-mentioned efficient image-text retrieval method based on multimodal feature fusion, comprising:

[0129] The image-text retrieval module is used to obtain input image-text retrieval data and extract image-text retrieval feature information from the image-text retrieval data;

[0130] A search judgment module is used to calculate the image and text search recognition parameters according to the image and text search feature information, and to judge whether the image and text search recognition parameters meet the preset conditions;

[0131] If the image-text retrieval recognition parameters do not meet the preset conditions, the image-text retrieval feature information is determined to not meet the quick retrieval conditions and is marked as fuzzy image-text feature information;

[0132] A planning resource module, used to calculate the planning retrieval system resource utilization data according to the fuzzy graphic feature information;

[0133] Fluctuation module, used to obtain and quickly retrieve system resource utilization fluctuation data;

[0134] An available resource module, used to obtain available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data;

[0135] An actual resource module, used for calculating actual retrieval system resource utilization data according to available retrieval system resource utilization data and planned retrieval system resource utilization data;

[0136] The duration module is used to obtain the retrieval duration based on the actual retrieval system resource utilization data.

[0137] As mentioned above, the image-text retrieval module is responsible for extracting image-text retrieval feature information from the input image-text retrieval data. It may include image processing algorithms and text processing algorithms for extracting feature representations of images and texts. These feature representations may include color histograms, text keywords, semantic vectors, etc. The retrieval judgment module calculates the image-text retrieval recognition parameters based on the extracted image-text retrieval feature information and determines whether it meets the preset conditions. If it meets the conditions, it can be quickly retrieved. If it does not meet the conditions, it will be marked as fuzzy image-text feature information, which helps to screen out image-text retrieval data that meets the requirements and improve the efficiency of subsequent processing. Based on the fuzzy image-text feature information, the planned resource module calculates the planned retrieval system resource utilization data. It can estimate the required system resources based on the preset resource allocation strategy and the characteristics of the retrieval task, and provide a reference for subsequent resource scheduling. The fluctuation module obtains the rapid retrieval system resource utilization fluctuation data for analyzing the fluctuation of system resource utilization. It can monitor the dynamic changes of system resources and provide a real-time reference basis for resource allocation and scheduling. Based on the rapid retrieval system resource utilization fluctuation data, the available resource module obtains the available retrieval system resources. Using data, it estimates the current available resources of the system by analyzing the fluctuations in system resource utilization, and provides guidance for subsequent resource allocation. The actual resource module calculates the actual retrieval system resource utilization data based on the available retrieval system resource utilization data and the planned retrieval system resource utilization data. It optimizes the resource allocation strategy based on the actual and expected conditions of the system resources to improve the resource utilization efficiency and performance of the system. Finally, based on the actual retrieval system resource utilization data, the duration module obtains the retrieval duration. It analyzes the utilization of system resources and estimates the time required for each retrieval task based on the resource utilization rate, providing users with accurate retrieval duration information. Through multimodal feature fusion and resource utilization optimization, the system can more accurately identify and process image and text retrieval requests, improving the efficiency and accuracy of retrieval. According to the actual resource utilization and system load, the system can dynamically adjust the resource allocation strategy to make more reasonable use of system resources and avoid resource waste and shortage. Through the monitoring and adjustment of the fluctuation module and the actual resource module, the system can better cope with the fluctuations and changes in system resource utilization and improve the stability and reliability of the system.

[0138] And, an efficient image and text retrieval terminal based on multimodal feature fusion, comprising:

[0139] one or more processors;

[0140] a storage device having one or more programs stored thereon;

[0141] When the one or more programs are executed by the one or more processors, the one or more processors implement the efficient image-text retrieval method based on multimodal feature fusion.

[0142] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. An efficient image-text retrieval method based on multimodal feature fusion, It is characterized in that include: Obtaining input image-text retrieval data, and extracting image-text retrieval feature information from the image-text retrieval data; Calculate the image and text retrieval recognition parameters according to the image and text retrieval feature information, and determine whether the image and text retrieval recognition parameters meet the preset conditions; If the image-text retrieval recognition parameters do not meet the preset conditions, the image-text retrieval feature information is determined to not meet the quick retrieval conditions and is marked as fuzzy image-text feature information; Calculate the plan to retrieve system resource utilization data based on fuzzy graphic feature information; Get quick retrieval of system resource utilization fluctuation data; Obtaining available retrieval system resource utilization data according to rapid retrieval system resource utilization fluctuation data; Calculate actual retrieval system resource utilization data based on available retrieval system resource utilization data and planned retrieval system resource utilization data; Obtain the search duration based on the actual search system resource utilization data; The step of obtaining the rapid retrieval system resource utilization fluctuation data comprises: Obtaining the image and text retrieval feature information corresponding to the retrieval identification parameters that meet the preset conditions, and marking it as clear image and text retrieval feature information; Obtaining the system resource utilization rate of the clear image and text retrieval feature information when performing image and text retrieval, and marking it as the rapid retrieval system resource utilization rate, wherein the rapid retrieval system resource utilization rate includes the historical rapid retrieval system resource utilization rate and the current rapid retrieval system resource utilization rate; Get quick retrieval system resource utilization fluctuation function; Inputting the historical rapid retrieval system resource utilization rate and the current rapid retrieval system resource utilization rate into the rapid retrieval system resource utilization fluctuation function, and marking the output result as the rapid retrieval system resource utilization fluctuation parameter to obtain the rapid retrieval system resource utilization fluctuation data; The step of obtaining available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data comprises: Get the current system resource utilization; Obtaining a rapid retrieval standard system resource utilization fluctuation parameter and a rapid retrieval standard system resource utilization rate; Get available retrieval system resource utilization function; The current system resource utilization, the quick retrieval system resource utilization fluctuation parameter, the quick retrieval standard system resource utilization fluctuation parameter and the quick retrieval standard system resource utilization are input into the available retrieval system resource utilization function, and the output result is marked as the available retrieval system resource utilization to obtain the available retrieval system resource utilization data.

2. According to the efficient image-text retrieval method based on multimodal feature fusion according to claim 1, It is characterized in that The step of obtaining input image-text retrieval data and extracting image-text retrieval feature information from the image-text retrieval data includes: Get the input image and text retrieval data; Extracting text retrieval feature information and image retrieval feature information from image-text retrieval data; The text retrieval feature information and the image retrieval feature information are aggregated into the image-text retrieval feature information.

3. According to the efficient image-text retrieval method based on multimodal feature fusion according to claim 1, It is characterized in that The step of calculating the image-text retrieval identification parameter according to the image-text retrieval feature information and judging whether the image-text retrieval identification parameter meets the preset conditions includes: Obtain corresponding text retrieval feature parameters and image retrieval feature parameters according to the image-text retrieval feature information; Obtaining text standard retrieval feature parameters and image standard retrieval feature parameters; Get the image and text retrieval recognition function; Inputting text retrieval feature parameters, image retrieval feature parameters, text standard retrieval feature parameters and image standard retrieval feature parameters into a graphic-text retrieval recognition function, and marking the output result as a graphic-text retrieval recognition parameter; Obtaining a standard image and text search and recognition evaluation interval, and determining whether the image and text search and recognition parameters are within the standard image and text search and recognition evaluation interval; If the image-text retrieval recognition parameter is within the image-text standard retrieval recognition evaluation interval, it is determined that the image-text retrieval feature information meets the fast retrieval condition; If the image-text retrieval recognition parameter is not within the image-text standard retrieval recognition evaluation interval, it is determined that the image-text retrieval feature information does not meet the quick retrieval conditions.

4. According to claim 1, the efficient image-text retrieval method based on multimodal feature fusion, It is characterized in that The step of calculating a plan to retrieve system resource utilization data according to the fuzzy graphic feature information comprises: Obtain corresponding image-text retrieval recognition parameters according to fuzzy image-text feature information; Obtaining standard image and text retrieval identification parameters and image and text standard retrieval system resource utilization; Get the plan to retrieve the system resource utilization function; The image-text retrieval identification parameters, image-text standard retrieval identification parameters and image-text standard retrieval system resource utilization are input into the planned retrieval system resource utilization function, and the output result is marked as the planned retrieval system resource utilization to obtain the planned retrieval system resource utilization data.

5. According to claim 1, the efficient image-text retrieval method based on multimodal feature fusion, It is characterized in that The step of calculating the actual retrieval system resource utilization data according to the available retrieval system resource utilization data and the planned retrieval system resource utilization data comprises: Obtaining the corresponding available retrieval system resource utilization rate according to the available retrieval system resource utilization data, and obtaining the corresponding planned retrieval system resource utilization rate according to the planned retrieval system resource utilization data; Get the actual retrieval volatility function; Input the available retrieval system resource utilization and the planned retrieval system resource utilization into the actual retrieval fluctuation function, and mark the output result as the actual retrieval fluctuation value; Determine whether the actual retrieval fluctuation value is greater than zero; If the actual search fluctuation value is less than or equal to zero, then the available search system resource utilization is determined to be the actual search system resource utilization data; If the actual retrieval fluctuation value is greater than zero, it is determined that the planned retrieval system resource utilization is the actual retrieval system resource utilization data.

6. The efficient image-text retrieval method based on multimodal feature fusion according to claim 1, It is characterized in that The step of obtaining the retrieval duration according to the actual retrieval system resource utilization data comprises: Acquire a search schedule, wherein the search schedule includes a plurality of search system resource utilization data and a search duration corresponding to each search system resource utilization data; The corresponding search duration is obtained in the search timetable according to the actual search system resource utilization data.

7. An efficient image-text retrieval system based on multimodal feature fusion, applied to the efficient image-text retrieval method based on multimodal feature fusion as claimed in any one of claims 1 to 6, It is characterized in that include: The image-text retrieval module is used to obtain input image-text retrieval data and extract image-text retrieval feature information from the image-text retrieval data; A search judgment module is used to calculate the image and text search recognition parameters according to the image and text search feature information, and to judge whether the image and text search recognition parameters meet the preset conditions; If the image-text retrieval recognition parameters do not meet the preset conditions, the image-text retrieval feature information is determined to not meet the quick retrieval conditions and is marked as fuzzy image-text feature information; A planning resource module, used to calculate the planning retrieval system resource utilization data according to the fuzzy graphic feature information; Fluctuation module, used to obtain and quickly retrieve system resource utilization fluctuation data; An available resource module, used to obtain available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data; An actual resource module, used for calculating actual retrieval system resource utilization data according to available retrieval system resource utilization data and planned retrieval system resource utilization data; Duration module, used to obtain retrieval duration according to actual retrieval system resource utilization data; The step of obtaining the rapid retrieval system resource utilization fluctuation data comprises: Obtaining the image and text retrieval feature information corresponding to the retrieval identification parameters that meet the preset conditions, and marking it as clear image and text retrieval feature information; Obtaining the system resource utilization rate of the clear image and text retrieval feature information when performing image and text retrieval, and marking it as the rapid retrieval system resource utilization rate, wherein the rapid retrieval system resource utilization rate includes the historical rapid retrieval system resource utilization rate and the current rapid retrieval system resource utilization rate; Get quick retrieval system resource utilization fluctuation function; Inputting the historical rapid retrieval system resource utilization rate and the current rapid retrieval system resource utilization rate into the rapid retrieval system resource utilization fluctuation function, and marking the output result as the rapid retrieval system resource utilization fluctuation parameter to obtain the rapid retrieval system resource utilization fluctuation data; The step of obtaining available retrieval system resource utilization data according to the rapid retrieval system resource utilization fluctuation data comprises: Get the current system resource utilization; Obtaining a rapid retrieval standard system resource utilization fluctuation parameter and a rapid retrieval standard system resource utilization rate; Get available retrieval system resource utilization function; The current system resource utilization, the quick retrieval system resource utilization fluctuation parameter, the quick retrieval standard system resource utilization fluctuation parameter and the quick retrieval standard system resource utilization are input into the available retrieval system resource utilization function, and the output result is marked as the available retrieval system resource utilization to obtain the available retrieval system resource utilization data.

8. An efficient image and text retrieval terminal based on multimodal feature fusion, It is characterized in that include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the efficient image-text retrieval method based on multimodal feature fusion as described in any one of claims 1 to 6.