Multi-modal distribution network hot-line work knowledge retrieval method and system

Abstract

The invention discloses a multi-modal distribution network hot-line work knowledge retrieval method and system. The method comprises the following steps: acquiring target distribution network hot-line work data; constructing an initial vector embedding model based on a Chine CLIP model, and training the initial vector embedding model based on the target distribution network live working data to obtain a target vector embedding model; inputting the target distribution network live working data into a target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vectors and the text vectors to obtain a knowledge retrieval database; and obtaining distribution network hot-line work query data, performing cross-modal retrieval on the distribution network hot-line work query data based on the knowledge retrieval database to obtain a cross-modal vector corresponding to the distribution network hot-line work query data, and converting the cross-modal vector into corresponding modal data. According to the invention, efficient joint query of multi-modal information can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent distribution network, and in particular to a multi-modal distribution network live working knowledge retrieval method and system. Background Art

[0002] With the continuous improvement of the intelligence and automation level of distribution networks, live operations in distribution networks have gradually shifted from manual operations to intelligent robot operations. However, intelligent robots still face many technical bottlenecks in key operating procedures, skill planning, and the enforcement of safety regulations. In the prior art, certain progress has been made in the automated retrieval methods for live operations in distribution networks through single-modal data support such as image recognition and text analysis. For example, some distribution network knowledge bases use computer vision technology to identify specific objects in distribution network scenarios, and link them with the corresponding operating procedure texts to form a preliminary retrieval system. However, the prior art relies on single-modal input and it is difficult to fully describe the rich information in multimodal data. In addition, the existing distribution network knowledge retrieval systems mostly use traditional databases as storage structures, which are difficult to meet real-time retrieval requirements and cannot adapt to the dynamic update requirements of data that changes with the operating environment.

[0003] In order to improve the accuracy and efficiency of knowledge retrieval in live distribution network operations, some studies in recent years have explored the application of multimodal learning, and tried to construct a more comprehensive knowledge expression method by integrating multimodal data such as images and texts. For example, multimodal alignment methods based on contrastive learning models have been proven to effectively embed multimodal information into a unified space, so that the similarity between images and texts can be quantified. However, when such methods are applied to live distribution network operations, there are still some unresolved technical problems, mainly including: 1) Insufficient integration of multimodal information: the existing system cannot achieve deep association between image and text data, resulting in retrieval results that are difficult to meet actual needs; 2) Low retrieval efficiency: the retrieval of image or text data alone lacks comprehensive processing methods, which reduces the response speed of the system; 3) Low knowledge matching accuracy: cross-modal data matching is difficult to meet the demand for high-precision knowledge in the live distribution network operation environment, affecting the accuracy and reliability of knowledge retrieval. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a multi-modal distribution network live working knowledge retrieval method and system.

[0005] In a first aspect, an embodiment of the present invention provides a multi-modal distribution network live working knowledge retrieval method, comprising:

[0006] Acquire target distribution network live working data, wherein the target distribution network live working data includes target image data consisting of a plurality of images and target text data consisting of a plurality of texts;

[0007] Building an initial vector embedding model based on the Chinese CLIP model, and training the initial vector embedding model based on the live working data of the target distribution network to obtain a target vector embedding model;

[0008] Inputting the target distribution network live operation data into the target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vector and the text vector to obtain a knowledge retrieval database;

[0009] Acquire distribution network live working query data, perform cross-modal retrieval on the distribution network live working query data based on the knowledge retrieval database, obtain a cross-modal vector corresponding to the distribution network live working query data, and convert the cross-modal vector into corresponding modal data, wherein the distribution network live working query data is image modal data or text modal data.

[0010] Preferably, the acquiring of live working data of the target distribution network includes:

[0011] Acquire initial distribution network live-line working data, wherein the initial distribution network live-line working data includes initial image data consisting of a plurality of images and initial text data consisting of a plurality of texts;

[0012] Performing image processing on the initial image data to obtain target image data, wherein the image processing includes resolution adjustment, noise removal, image enhancement and labeling;

[0013] Performing text processing on the initial text data to obtain target text data, wherein the text processing includes word segmentation, stop word removal and labeling;

[0014] Target distribution network live working data is obtained based on the target image data and the target text data.

[0015] Preferably, the training of the initial vector embedding model based on the target distribution network live working data to obtain the target vector embedding model includes:

[0016] Pairing the target image data with the target text data to obtain a comparative learning data set consisting of N image-text sample pairs;

[0017] Input the contrastive learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, wherein each element in the similarity matrix is ​​the similarity of an image-text sample pair;

[0018] Based on the similarity matrix, constructing a contrast loss function with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs;

[0019] The initial vector embedding model is iteratively optimized based on the contrast loss function to obtain a target vector embedding model.

[0020] Preferably, the comparative learning data set is input into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, including:

[0021] Inputting the N images of the contrastive learning dataset into the image encoder in the initial vector embedding model for vectorization embedding respectively to obtain N image vectors;

[0022] Inputting N texts of the contrastive learning dataset into the text encoder in the initial vector embedding model for vectorization embedding respectively to obtain N text vectors;

[0023] Pair the N image vectors with the N text vectors in pairs to obtain an image-text vector pair matrix of size N*N;

[0024] The similarity of each image-text vector pair in the image-text vector pair matrix is ​​calculated to obtain an N*N similarity matrix.

[0025] Preferably, the constructing of a contrast loss function based on the similarity matrix with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs comprises:

[0026] The contrast loss function is characterized by the following formula:

[0027]

[0028] in, represents the contrast loss function, N represents the number of image-text sample pairs, represents the fraction corresponding to the i-th image, represents the fraction corresponding to the i-th text, v i represents the image vector corresponding to the i-th image, t i represents the text vector corresponding to the i-th text, sim(·) represents the similarity function, and τ represents the temperature parameter.

[0029] Preferably, the storing the image vector and the text vector to obtain a knowledge retrieval database comprises:

[0030] The image vector and the text vector are stored in a Milvus database respectively, and the Milvus database is characterized as a knowledge retrieval database.

[0031] Preferably, the acquiring of the distribution network live working query data, performing a cross-modal search on the distribution network live working query data based on the knowledge retrieval database, obtaining a cross-modal vector corresponding to the distribution network live working query data, and converting the cross-modal vector into corresponding modal data, includes:

[0032] Acquire distribution network live-working query data, and input the distribution network live-working query data into the target vector embedding model for vectorized embedding to obtain a query vector corresponding to the distribution network live-working query data;

[0033] A cross-modal vector having the highest similarity to the query vector is retrieved in the knowledge retrieval database, and the cross-modal vector is converted into corresponding modal data.

[0034] In a second aspect, an embodiment of the present invention provides a multi-modal distribution network live working knowledge retrieval system, comprising:

[0035] A data acquisition module, used to acquire target distribution network live working data, wherein the target distribution network live working data includes target image data consisting of a plurality of images and target text data consisting of a plurality of texts;

[0036] A model building module, used to build an initial vector embedding model based on the Chinese CLIP model, and train the initial vector embedding model based on the live working data of the target distribution network to obtain a target vector embedding model;

[0037] A database construction module, used for inputting the target distribution network live operation data into the target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vector and the text vector to obtain a knowledge retrieval database;

[0038] A cross-modal retrieval module is used to obtain distribution network live working query data, perform cross-modal retrieval on the distribution network live working query data based on the knowledge retrieval database, obtain the cross-modal vector corresponding to the distribution network live working query data, and convert the cross-modal vector into corresponding modal data, wherein the distribution network live working query data is image modal data or text modal data.

[0039] Preferably, the model building module comprises:

[0040] A pairing unit, used to pair the target image data with the target text data to obtain a comparative learning data set consisting of N image-text sample pairs;

[0041] A similarity matrix construction unit is used to input the comparative learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, wherein each element in the similarity matrix is ​​the similarity of an image-text sample pair;

[0042] A contrast loss function construction unit, used to construct a contrast loss function based on the similarity matrix with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs;

[0043] An iterative optimization unit is used to iteratively optimize the initial vector embedding model based on the contrast loss function to obtain a target vector embedding model.

[0044] Preferably, the database construction module includes:

[0045] The vector storage unit is used to store the image vector and the text vector in the Milvus database respectively, and characterize the Milvus database as a knowledge retrieval database.

[0046] Compared with the prior art, the multimodal distribution network live working knowledge retrieval method and system of the embodiment of the present invention has the following beneficial effects: through the comparative learning method, images and texts are mapped to the same semantic space, which not only realizes cross-modal data association and semantic matching, but also realizes the deep fusion of multimodal data, and can directly compare the similarity of images and texts in a unified semantic space, overcomes the limitations of traditional single-modal processing, and provides superior cross-modal retrieval capabilities; by introducing the loss function of comparative learning, the semantic consistency of images and texts in the same vector space is optimized; the Milvus database specially optimized for high-dimensional vector data is adopted, and the Milvus database supports GPU acceleration and distributed deployment, can process large-scale data and return query results within milliseconds, which significantly improves the retrieval efficiency, thereby quickly providing the required knowledge support in emergency scenarios of live working and meeting the requirements of real-time and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a multi-modal distribution network live working knowledge retrieval method according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of a process for constructing a target vector embedding model according to an embodiment of the present invention;

[0049] Figure 3is a schematic diagram of constructing an image-text vector pair matrix according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a process of cross-modal retrieval according to an embodiment of the present invention;

[0051] Figure 5 It is a structural schematic diagram of a multi-modal distribution network live working knowledge retrieval system according to an embodiment of the present invention;

[0052] Figure 6 It is a structural diagram of a model building module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0054] In the description of the present invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0055] like Figure 1 As shown, an embodiment of the present invention provides a multi-modal distribution network live working knowledge retrieval method, comprising the steps of:

[0056] S1. Obtaining live operation data of target distribution network;

[0057] Specifically, step S1 includes:

[0058] 1) Obtaining initial distribution network live operation data;

[0059] The initial distribution network live operation data includes initial image data consisting of a number of images and initial text data consisting of a number of texts. Specifically, the initial image data includes photos of normal equipment, photos of the working environment, and photos of faulty equipment, and the initial text data includes operating specifications and fault diagnosis records.

[0060] 2) performing image processing on the initial image data to obtain target image data;

[0061] Image processing includes resolution adjustment, noise removal, image enhancement, and labeling. The initial image data is subjected to resolution adjustment, noise removal, image enhancement, and other operations to improve image quality and model training effects. At the same time, the initial image data is labeled to enhance the model's understanding of semantics. Labeling includes annotating device types and fault types.

[0062] 3) Performing text processing on the initial text data to obtain target text data;

[0063] Text processing includes word segmentation, stop word removal, and tagging. The initial text data is subjected to word segmentation, stop word removal, tagging, etc. to ensure the consistency of the input data format. Specifically, this embodiment performs standardization processing on technical terms and device names to ensure the accuracy of the target text data during retrieval.

[0064] 4) Obtain target distribution network live operation data based on target image data and target text data.

[0065] Specifically, the target distribution network live operation data includes target image data consisting of a plurality of images and target text data consisting of a plurality of texts.

[0066] S2. Build an initial vector embedding model based on the Chinese CLIP model, and train the initial vector embedding model based on the live operation data of the target distribution network to obtain a target vector embedding model;

[0067] Specifically, Figure 2 As shown, step S2 includes:

[0068] S201, pairing the target image data and the target text data to obtain a comparative learning data set consisting of N image-text sample pairs;

[0069] The images and texts associated with the target distribution network live operation data are paired to form sample pairs, forming a comparative learning dataset consisting of N image-text sample pairs.

[0070] S202, inputting the comparative learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N;

[0071] Each pair of images and texts are embedded into the same vector space using the initial vector embedding model built based on the Chinese CLIP model. Images are feature extracted through the image encoder, while texts are encoded through the text encoder, and finally the corresponding vector representation is generated.

[0072] Specifically, step S202 includes:

[0073] 1) Input the N images of the comparative learning dataset into the image encoder in the initial vector embedding model for vectorization embedding, and obtain N image vectors;

[0074] The image encoder of this embodiment is a VIT deep learning framework, which uses VIT to vectorize and embed N images respectively to obtain N image vectors.

[0075] 2) Input the N texts of the comparative learning dataset into the text encoder in the initial vector embedding model for vectorization embedding respectively to obtain N text vectors;

[0076] The text encoder in this embodiment is Bert, and N texts are respectively vectorized and embedded through Bert to obtain N text vectors.

[0077] 3) Pair the N image vectors and the N text vectors in pairs to obtain an N*N image-text vector pair matrix;

[0078] Figure 3 A schematic diagram of the image-text vector pair matrix obtained based on the above steps 1) to 3) is shown, T1 to T N Represents N text vectors, I1~I N Represents N image vectors.

[0079] 4) Calculate the similarity of each image-text vector pair in the image-text vector pair matrix to obtain an N*N similarity matrix.

[0080] Each element in the similarity matrix is ​​the similarity of an image-text sample pair. This embodiment uses cosine similarity, which is defined as:

[0081]

[0082] Among them, sim(a,b) represents the cosine similarity function.

[0083] S203, based on the similarity matrix, constructing a contrast loss function with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs;

[0084] The contrast loss function is introduced to make the vector representations of images and texts semantically closer, that is, related images and texts have higher similarity, while unrelated images and texts have lower similarity. When calculating the contrast loss, the CLIP model considers that only the pairs of images and texts on the diagonal in the similarity matrix are positive samples, and the rest are negative samples, that is, the similarity of similar pairs is maximized, and the similarity of dissimilar pairs is minimized.

[0085] Specifically, the following formula is used to characterize the contrast loss function:

[0086]

[0087] in, represents the contrast loss function, N represents the number of image-text sample pairs, represents the fraction corresponding to the i-th image, represents the fraction corresponding to the i-th text, v i represents the image vector corresponding to the i-th image, t i represents the text vector corresponding to the ith text, sim(·) represents the similarity function, and τ represents the temperature parameter used to control the smoothness of the distribution. This embodiment adopts the cosine similarity function, and its definition is referred to above and will not be repeated here.

[0088] S204. Iteratively optimize the initial vector embedding model based on the contrast loss function to obtain a target vector embedding model.

[0089] During the iterative optimization process, the initial vector embedding model minimizes the contrast loss function so that the distance between matching image-text sample pairs in the feature space is closer, while the distance between mismatching image-text sample pairs is farther, thereby achieving alignment between images and texts and obtaining the target vector embedding model.

[0090] The initial vector embedding model is iteratively optimized based on contrast loss to gradually improve the recognition ability of the initial vector embedding model on multimodal distribution network live working knowledge. After training, the target vector embedding model can efficiently associate image data and text data in the same vector space, thereby supporting subsequent cross-modal retrieval needs.

[0091] S3, inputting the target distribution network live operation data into the target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vector and the text vector to obtain a knowledge retrieval database;

[0092] Specifically, the image vector and the text vector are stored in the Milvus database respectively, and the Milvus database is characterized as a knowledge retrieval database. The Milvus database is an efficient vector database designed for processing large-scale vector data and can support high-performance similarity retrieval.

[0093] Furthermore, this embodiment stores the image vector and text vector in the Milvus database respectively, and creates an index to improve the retrieval efficiency. At the same time, the stored image vector and text vector are optimized by using the Milvus indexing function, including IVF (inverted file) index and HNSW (hierarchical navigation small world graph) index, to ensure the retrieval speed and accuracy during query.

[0094] S4. Obtain the distribution network live operation query data, perform cross-modal retrieval on the distribution network live operation query data based on the knowledge retrieval database, obtain the cross-modal vector corresponding to the distribution network live operation query data, and convert the cross-modal vector into corresponding modal data.

[0095] Specifically, Figure 4 As shown, step S4 includes:

[0096] S401, obtaining distribution network live operation query data, and inputting the distribution network live operation query data into the target vector embedding model for vectorized embedding, to obtain a query vector corresponding to the distribution network live operation query data;

[0097] The query data of live-line operation of distribution network is image modal data or text modal data. That is to say, if the query data of live-line operation of distribution network is image modal data, the corresponding query vector is an image vector; if the query data of live-line operation of distribution network is text modal data, the corresponding query vector is a text vector.

[0098] S402: Retrieve the cross-modal vector with the highest similarity to the query vector in the knowledge retrieval database, and convert the cross-modal vector into corresponding modal data.

[0099] If the query vector is an image vector, the text vector with the highest similarity to the image vector is retrieved in the knowledge retrieval database, and the text vector is converted into corresponding text data; if the query vector is a text vector, the image vector with the highest similarity to the text vector is retrieved in the knowledge retrieval database, and the image vector is converted into corresponding image data.

[0100] The embodiment of the present invention provides a multimodal distribution network live working knowledge retrieval method, which maps images and texts to the same semantic space through a contrastive learning method, realizes cross-modal data association and semantic matching, and realizes deep fusion of multimodal data. It can directly compare the similarity of images and texts in a unified semantic space, overcomes the limitations of traditional single-modal processing, and provides superior cross-modal retrieval capabilities; by introducing a contrastive learning loss function, it optimizes the semantic consistency of images and texts in the same vector space; adopts a Milvus database specially optimized for high-dimensional vector data, and the Milvus database supports GPU acceleration and distributed deployment, can process large-scale data and return query results within milliseconds, significantly improving retrieval efficiency, thereby quickly providing the required knowledge support in emergency scenarios of live working and meeting the requirements of real-time and reliability.

[0101] Based on the above multi-modal distribution network live working knowledge retrieval method, Figure 5 As shown, an embodiment of the present invention provides a multi-modal distribution network live working knowledge retrieval system, including:

[0102] The data acquisition module 1 is used to acquire the target distribution network live working data, wherein the target distribution network live working data includes target image data composed of a plurality of images and target text data composed of a plurality of texts;

[0103] Model building module 2, used to build an initial vector embedding model based on the Chinese CLIP model, and train the initial vector embedding model based on the target distribution network live operation data to obtain a target vector embedding model;

[0104] Specifically, Figure 6 As shown, the model building module 2 includes:

[0105] A pairing unit 21 is used to pair the target image data with the target text data to obtain a comparative learning data set consisting of N image-text sample pairs;

[0106] A similarity matrix construction unit 22 is used to input the comparative learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, wherein each element in the similarity matrix is ​​the similarity of an image-text sample pair;

[0107] A contrast loss function construction unit 23, for constructing a contrast loss function based on a similarity matrix with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs;

[0108] The iterative optimization unit 24 is used to iteratively optimize the initial vector embedding model based on the contrast loss function to obtain a target vector embedding model.

[0109] Database construction module 3, used for inputting the target distribution network live operation data into the target vector embedding model for vectorization embedding to obtain the image vector corresponding to each image and the text vector corresponding to each text, and storing the image vector and the text vector to obtain a knowledge retrieval database;

[0110] Specifically, the database construction module 3 includes:

[0111] The vector storage unit is used to store the image vector and the text vector respectively in the Milvus database, and represent the Milvus database as a knowledge retrieval database.

[0112] The cross-modal retrieval module 4 is used to obtain the distribution network live operation query data, perform cross-modal retrieval on the distribution network live operation query data based on the knowledge retrieval database, obtain the cross-modal vector corresponding to the distribution network live operation query data, and convert the cross-modal vector into corresponding modal data, wherein the distribution network live operation query data is image modal data or text modal data.

[0113] It should be noted that each module in the above-mentioned multimodal distribution network live working knowledge retrieval system can be fully or partially implemented by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. For the specific definition of a multimodal distribution network live working knowledge retrieval system, please refer to the definition of a multimodal distribution network live working knowledge retrieval method in the above text. The two have the same functions and effects, which will not be repeated here.

[0114] In summary, the embodiment of the present invention provides a multimodal distribution network live working knowledge retrieval method and system, which maps images and texts to the same semantic space through a comparative learning method, realizes cross-modal data association and semantic matching, and realizes deep fusion of multimodal data. It can directly compare the similarity of images and texts in a unified semantic space, overcomes the limitations of traditional single-modal processing, and provides superior cross-modal retrieval capabilities; by introducing a comparative learning loss function, it optimizes the semantic consistency of images and texts in the same vector space; adopts a Milvus database specially optimized for high-dimensional vector data, which supports GPU acceleration and distributed deployment, can process large-scale data and return query results within milliseconds, significantly improving the retrieval efficiency, thereby quickly providing the required knowledge support in emergency scenarios of live working and meeting the requirements of real-time and reliability.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-modal distribution network live working knowledge retrieval method, characterized in that: include: Acquire target distribution network live working data, wherein the target distribution network live working data includes target image data consisting of a plurality of images and target text data consisting of a plurality of texts; Building an initial vector embedding model based on the Chinese CLIP model, and training the initial vector embedding model based on the live working data of the target distribution network to obtain a target vector embedding model; Inputting the target distribution network live operation data into the target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vector and the text vector to obtain a knowledge retrieval database; Acquire distribution network live working query data, perform cross-modal retrieval on the distribution network live working query data based on the knowledge retrieval database, obtain a cross-modal vector corresponding to the distribution network live working query data, and convert the cross-modal vector into corresponding modal data, wherein the distribution network live working query data is image modal data or text modal data.

2. The multi-modal distribution network live working knowledge retrieval method according to claim 1 is characterized in that: The step of obtaining live working data of a target distribution network includes: Acquire initial distribution network live-line working data, wherein the initial distribution network live-line working data includes initial image data consisting of a plurality of images and initial text data consisting of a plurality of texts; Performing image processing on the initial image data to obtain target image data, wherein the image processing includes resolution adjustment, noise removal, image enhancement and labeling; Performing text processing on the initial text data to obtain target text data, wherein the text processing includes word segmentation, stop word removal and labeling; Target distribution network live working data is obtained based on the target image data and the target text data.

3. The multi-modal distribution network live working knowledge retrieval method according to claim 1 is characterized in that: The training of the initial vector embedding model based on the target distribution network live working data to obtain a target vector embedding model includes: Pairing the target image data with the target text data to obtain a comparative learning data set consisting of N image-text sample pairs; Input the contrastive learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, wherein each element in the similarity matrix is ​​the similarity of an image-text sample pair; Based on the similarity matrix, constructing a contrast loss function with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs; The initial vector embedding model is iteratively optimized based on the contrast loss function to obtain a target vector embedding model.

4. The multi-modal distribution network live working knowledge retrieval method according to claim 3 is characterized in that: The step of inputting the contrastive learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, including: Inputting the N images of the contrastive learning dataset into the image encoder in the initial vector embedding model for vectorization embedding respectively to obtain N image vectors; Inputting N texts of the contrastive learning dataset into the text encoder in the initial vector embedding model for vectorization embedding respectively to obtain N text vectors; Pair the N image vectors with the N text vectors in pairs to obtain an image-text vector pair matrix of size N*N; The similarity of each image-text vector pair in the image-text vector pair matrix is ​​calculated to obtain an N*N similarity matrix.

5. The multi-modal distribution network live working knowledge retrieval method according to claim 3 is characterized in that: The method constructs a contrast loss function based on the similarity matrix with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs, including: The contrast loss function is characterized by the following formula: in, represents the contrast loss function, N represents the number of image-text sample pairs, represents the fraction corresponding to the i-th image, represents the fraction corresponding to the i-th text, v i represents the image vector corresponding to the i-th image, t i represents the text vector corresponding to the i-th text, sim(·) represents the similarity function, and τ represents the temperature parameter.

6. The multi-modal distribution network live working knowledge retrieval method according to claim 1 is characterized in that: The storing of the image vector and the text vector to obtain a knowledge retrieval database comprises: The image vector and the text vector are stored in a Milvus database respectively, and the Milvus database is characterized as a knowledge retrieval database.

7. The multi-modal distribution network live working knowledge retrieval method according to claim 1 is characterized in that: The acquiring of the distribution network live working query data, performing a cross-modal search on the distribution network live working query data based on the knowledge retrieval database, obtaining a cross-modal vector corresponding to the distribution network live working query data, and converting the cross-modal vector into corresponding modal data, includes: Acquire distribution network live-working query data, and input the distribution network live-working query data into the target vector embedding model for vectorized embedding to obtain a query vector corresponding to the distribution network live-working query data; A cross-modal vector having the highest similarity to the query vector is retrieved in the knowledge retrieval database, and the cross-modal vector is converted into corresponding modal data.

8. A multi-modal distribution network live working knowledge retrieval system, characterized in that: include: A data acquisition module, used to acquire target distribution network live working data, wherein the target distribution network live working data includes target image data consisting of a plurality of images and target text data consisting of a plurality of texts; A model building module, used to build an initial vector embedding model based on the Chinese CLIP model, and train the initial vector embedding model based on the live working data of the target distribution network to obtain a target vector embedding model; A database construction module, used for inputting the target distribution network live operation data into the target vector embedding model for vectorization embedding to obtain an image vector corresponding to each image and a text vector corresponding to each text, and storing the image vector and the text vector to obtain a knowledge retrieval database; A cross-modal retrieval module is used to obtain distribution network live working query data, perform cross-modal retrieval on the distribution network live working query data based on the knowledge retrieval database, obtain the cross-modal vector corresponding to the distribution network live working query data, and convert the cross-modal vector into corresponding modal data, wherein the distribution network live working query data is image modal data or text modal data.

9. The multi-modal distribution network live working knowledge retrieval system according to claim 8, characterized in that: The model building module comprises: A pairing unit, used to pair the target image data with the target text data to obtain a comparative learning data set consisting of N image-text sample pairs; A similarity matrix construction unit is used to input the comparative learning data set into the initial vector embedding model for vectorization embedding to obtain a similarity matrix of size N*N, wherein each element in the similarity matrix is ​​the similarity of an image-text sample pair; A contrast loss function construction unit, used to construct a contrast loss function based on the similarity matrix with the goal of maximizing the similarity of similar image-text sample pairs and minimizing the similarity of dissimilar image-text sample pairs; An iterative optimization unit is used to iteratively optimize the initial vector embedding model based on the contrast loss function to obtain a target vector embedding model.

10. The multi-modal distribution network live working knowledge retrieval system according to claim 8, characterized in that: The database construction module includes: The vector storage unit is used to store the image vector and the text vector in a Milvus database respectively, and characterize the Milvus database as a knowledge retrieval database.

Images