Operation decision-making method, device and system of operation inspection robot and medium

Abstract

The invention provides an operation decision-making method, device and system of an operation inspection robot and a medium. According to the implementation scheme, a plurality of positive sample pairs and a plurality of negative sample pairs are constructed; training a cross-modal semantic alignment model based on the plurality of positive sample pairs and the plurality of negative sample pairs, performing semantic alignment on each positive sample pair by adopting the trained cross-modal semantic alignment model to obtain a knowledge vector of semantic alignment, and storing the knowledge vector into a multi-modal operation inspection knowledge base; in response to an operation request text for the operation inspection robot, performing semantic alignment on the operation request text based on a cross-modal semantic alignment model to obtain a semantic-aligned operation request text vector; searching a target knowledge vector similar to the job request text vector in a multi-modal operation inspection knowledge base; and based on the target knowledge vector and the operation request text, generating an operation action instruction of the operation inspection robot. According to the embodiment of the invention, the operation decision accuracy of the operation inspection robot can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operation decision-making method, device, system and medium for an operation inspection robot. Background Art

[0002] With the development of smart grids, industrial automation, and intelligent operations and maintenance (O&M), operational robots are increasingly being used in power operations and maintenance, equipment maintenance, and fault diagnosis. The operating environment of power equipment is complex and volatile, and current intelligent O&M systems rely primarily on pre-set rules and single-modal retrieval methods. Consequently, they struggle to meet the demands for efficient and accurate power O&M in key areas such as fault diagnosis, operation planning, and decision support, as well as the need for high-precision operational decision retrieval in live working environments.

[0003] In addition, the operation and inspection process involves multimodal information such as text, images, and videos, but the existing operation and inspection robot knowledge base is mainly based on structured data, which makes it difficult to achieve deep correlation between image, text and other data, resulting in difficulty for inspection robots to accurately retrieve or match key information in actual operations.

[0004] Therefore, there is an urgent need for a job decision method that can improve retrieval accuracy in job decision making. Summary of the Invention

[0005] The present invention provides an operation decision-making method, device, system and medium for an operation and inspection robot, which can solve at least one of the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides an operation decision-making method for an inspection robot, comprising:

[0007] Based on the historical job request text and historical job images of the operation and inspection robot, multiple positive sample pairs and multiple negative sample pairs are constructed;

[0008] Based on the multiple positive sample pairs and the multiple negative sample pairs, a cross-modal semantic alignment model is trained, and each of the positive sample pairs is semantically aligned using the trained cross-modal semantic alignment model to obtain a semantically aligned knowledge vector, and the knowledge vector is stored in a multimodal operation and inspection knowledge base;

[0009] In response to a job request text for the operation and inspection robot, semantically aligning the job request text based on the cross-modal semantic alignment model to obtain a semantically aligned job request text vector;

[0010] Retrieving a target knowledge vector similar to the job request text vector in the multimodal operation and inspection knowledge base;

[0011] Based on the target knowledge vector and the operation request text, an operation action instruction of the operation and inspection robot is generated.

[0012] In a second aspect, an embodiment of the present invention further provides an operation decision-making device for an inspection robot, comprising:

[0013] A construction module is used to construct multiple positive sample pairs and multiple negative sample pairs based on the historical job request text and historical job images of the operation and inspection robot;

[0014] A training module is configured to train a cross-modal semantic alignment model based on the multiple positive sample pairs and the multiple negative sample pairs, and perform semantic alignment on each of the positive sample pairs using the trained cross-modal semantic alignment model to obtain a semantically aligned knowledge vector, and store the knowledge vector in a multimodal operation and inspection knowledge base;

[0015] a semantic alignment module, configured to respond to a job request text for the operation and inspection robot and perform semantic alignment on the job request text based on the cross-modal semantic alignment model to obtain a semantically aligned job request text vector;

[0016] A retrieval module, configured to retrieve a target knowledge vector similar to the job request text vector in the multimodal operation and inspection knowledge base;

[0017] A generation module is used to generate an operation action instruction of the operation and inspection robot based on the target knowledge vector and the operation request text.

[0018] In a third aspect, an embodiment of the present invention further provides an operation decision system for an inspection robot, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods in the embodiments of the present invention.

[0022] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute any one of the methods in the embodiments of the present invention.

[0023] By adopting the technical solution of the present invention, a cross-modal semantic alignment model is trained by constructing positive and negative sample pairs based on the historical job request texts and historical job images of the operation and maintenance robot. The positive sample pairs are semantically aligned based on the trained cross-modal semantic alignment model to obtain a semantically aligned knowledge vector, and the knowledge vector is stored in a multimodal operation and maintenance knowledge base. The operation request texts for the operation and maintenance robot are semantically aligned using the cross-modal semantic alignment model to obtain a semantically aligned operation request text vector. In the multimodal operation and maintenance knowledge base, a knowledge vector similar to the operation request text vector is retrieved as the target knowledge vector. Finally, based on the target knowledge vector and the current operation request text, an operation action instruction suitable for the current scenario is generated to guide the operation and maintenance robot to perform the operation and maintenance operation, thereby improving the accuracy of its operation decision-making.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention.

[0026] Figure 1 This is a flow chart of an operation decision-making method for an inspection robot according to an embodiment of the present invention;

[0027] Figure 2 This is a sub-flowchart of the operation decision-making method of the operation inspection robot according to one embodiment of the present invention;

[0028] Figure 3 is another sub-flowchart of the operation decision-making method of the operation inspection robot according to one embodiment of the present invention;

[0029] Figure 4 This is a structural block diagram of the operation decision-making device of the inspection robot according to one embodiment of the present invention.

[0030] Figure 5 This is a structural block diagram of a training module in an operation decision-making device for an operation and inspection robot according to an embodiment of the present invention;

[0031] Figure 6 This is a structural block diagram of a target loss function determination unit in an operation decision-making device of an operation and inspection robot according to an embodiment of the present invention;

[0032] Figure 7 This is a structural block diagram of a retrieval module in an operation decision-making device of an inspection robot according to an embodiment of the present invention;

[0033] Figure 8 This is a structural block diagram of the third determination unit in the operation decision-making device of the operation inspection robot according to one embodiment of the present invention;

[0034] Figure 9 This is a structural block diagram of a generation module in an operation decision-making device of an inspection robot according to an embodiment of the present invention;

[0035] Figure 10 This is another structural block diagram of the operation decision-making device of the operation inspection robot according to one embodiment of the present invention;

[0036] Figure 11 is a block diagram of an electronic device for implementing the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, and various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] Figure 1 FIG. 1 is a flow chart of an operation decision-making method of an inspection robot according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.

[0039] S110 , constructing multiple positive sample pairs and multiple negative sample pairs based on the historical operation request texts and historical operation images of the operation and inspection robot.

[0040] Among them, historical job request text refers to various data with text data type during the previous operation of the operation and maintenance robot. This data is recorded and uploaded by humans or the system and is used to describe specific operation tasks. Historical operation images refer to various data with image or video data type during the actual operation and maintenance tasks. A positive sample pair refers to a sample pair in the form of "image-text" pairing consisting of a historical job request text and its corresponding one or more historical job images, that is, a positive sample pair includes a matching job text and one or more job images. A negative sample pair refers to all sample pairs consisting of a job text and one or more job images during the operation of the operation and maintenance robot.

[0041] Specifically, during live operations on distribution network maintenance, the maintenance robot can collect multiple types of operation data in various live operation scenarios through the multimodal sensors (such as cameras and infrared thermal imagers), operation logs, environmental monitoring equipment, and other means. The operation data of the maintenance robot's live operations on distribution network maintenance together constitute historical operation request texts and historical operation images. Among them, historical operation images include image data such as on-site equipment photos, environmental photos, equipment failure pictures, and frame image data obtained by image frame extraction and processing from video data such as standardized operation process videos; historical operation request texts include text data such as operation operation specifications, safety regulations, and fault diagnosis cases.

[0042] Power operation and maintenance operations involve a large amount of multimodal information such as images, text, and videos. By processing multimodal data information, several positive sample pairs in the form of "image-text" sample pairs are obtained to facilitate deep correlation of multimodal data. This allows the operation and maintenance robot to find the most matching image based on the query text, or obtain relevant text information based on the image, thereby accurately matching key information in actual operations.

[0043] Specifically, in one embodiment, metadata tags are added to each type of historical job image or contextual information is extracted (e.g., the image's source device, shooting time, geographic location, associated number, etc.). Typically, this information is automatically recorded during the image acquisition system or video frame extraction process. For example, the frame image extracted from the video will retain its original timestamp to facilitate alignment with the event record in the historical job request text. At the same time, key fields in the historical job request text are identified, especially key fields such as time information, device number, operation behavior, and event description.

[0044] Then, based on time alignment and device ID alignment, a multi-condition matching algorithm (such as hash matching, fuzzy matching, and rule-based filtering) is used to pair the image with the text. For example, if an image is taken from a camera of a "D1 circuit breaker" and the capture time is "10 seconds before and after 10:00 on June 5, 2024," and the text record mentions the tripping event of the D1 circuit breaker within the same time period, it can be determined that the two are highly correlated and constitute an "image-text" pair (a positive sample pair).

[0045] In addition to the aforementioned pairing methods, a semantic matching model can be used to score candidate text or images, selecting the text or image with the highest score (i.e., the closest semantic match) for pairing, thereby improving pairing accuracy. The semantic matching model can be Contrastive Language-Image Pretraining (CLIP) or the Twin Towers model for image-text matching.

[0046] Specifically, in some embodiments, S110 includes the following steps:

[0047] The video data is converted into a plurality of frame image data by intercepting frames at fixed intervals, and the frame image data and the original image data are used as the working image samples;

[0048] The job image samples and the job text samples are preprocessed respectively to obtain historical job request texts and historical job images.

[0049] Among them, frame image data refers to a set of frame images extracted from video data at a fixed time interval or fixed frame interval; original image data refers to various data with image data type in the operation and maintenance operation of the operation and maintenance robot; operation text samples refer to various historical operation request texts with text data type in the operation and maintenance operation of the operation and maintenance robot.

[0050] For example, video processing tools can be used to extract image frames from video data at fixed frame intervals to obtain multiple frames of image data. Video combing tools can utilize the Open Source Computer Vision Library (OpenCV) or the Fast Forward Moving Picture Experts Group (FFmpeg). By converting video data into multiple frames of image data using a specific method, a richer sample size can be provided for subsequent positive sample pair construction.

[0051] Specifically, the Convolutional Networks for Biomedical Image Segmentation (U-Net), a convolutional neural network with an encoder-decoder architecture, has demonstrated excellent performance in image semantic segmentation tasks. By using labeled clean and noisy image pairs during the training phase, the network learns to distinguish between the device itself and noise. During the decoding phase, skip connections are used to preserve high-resolution image details, resulting in cleaner output images without losing important structural information.

[0052] First, the original image data and frame image data are preprocessed, such as resizing all images to a uniform size and normalizing their pixel values ​​to scale them to the range [0, 1] or [-1, 1]. Second, the preprocessed images are fed into a U-Net network to generate segmentation masks. Based on the segmentation masks, rain, fog, and electromagnetic interference noise areas are removed from the original image data and frame image data, resulting in preliminarily processed images. The U-Net achieves fine-scale image noise removal, preserving image structural features while effectively removing external interference, thereby improving image clarity and usability. Finally, the images after removing rain, fog, and noise are fed into a pretrained Generative Adversarial Network (GAN) to generate images with enhanced device texture details in low-light scenarios. This image is then used as the historical operation image. The introduction of the GAN not only improves the overall image brightness but also restores the complex surface textures and key recognition features of the image, thereby improving the robustness and recognition rate of subsequent models in complex environments.

[0053] In addition, for operation text samples, the topological relationship between devices can be associated by building a power domain dictionary and using the Chinese bidirectional encoder representations from transformers (BERT) model for entity recognition, thereby enhancing the accuracy of the operation and maintenance robot's semantic understanding.

[0054] Specifically, the job text samples are cleaned and segmented to remove useless symbols, spaces, and special characters. The cleaned and segmented text data is input into the pre-trained BERT model, which outputs the entity label corresponding to each word in the text data. Entities are extracted from the text data based on the labels output by the BERT model. The topological relationship between devices can be determined through information such as the design drawings and network topology diagrams of the power system. By constructing a database of device topological relationships, the entities and the topological relationships between them are stored in the database as historical job request text. Combining entities and device topological relationships to perform semantic analysis on text data can improve the accuracy of semantic understanding of the operation and maintenance robot.

[0055] S120: Based on the multiple positive sample pairs and the multiple negative sample pairs, a cross-modal semantic alignment model is trained, and the trained cross-modal semantic alignment model is used to semantically align each of the positive sample pairs to obtain a semantically aligned knowledge vector, and the knowledge vector is stored in a multimodal operation and inspection knowledge base.

[0056] Among them, the cross-modal semantic alignment model aims to process data of different modalities (such as text and images) and map these different modal data into the same vector space, so that different modal data under the same semantic concept are close to each other in the vector space, thereby achieving semantic alignment between different modal data.

[0057] Specifically, in one embodiment, Figure 2 As shown, S120 includes the following steps:

[0058] S121: Input the multiple positive sample pairs and the multiple negative sample pairs into the cross-modal semantic alignment model, respectively, to obtain semantically aligned feature vectors of each positive sample pair and each negative sample pair, wherein the positive sample pair includes a sample pair consisting of a job text and one or more job images that match each other, and the negative sample pair includes a sample pair consisting of a job text and one or more job images that do not match each other;

[0059] S122, determining a target loss function based on the feature vectors of each of the positive sample pairs and the feature vectors of each of the negative sample pairs;

[0060] S123: Adjust the model parameters of the cross-modal semantic alignment model based on the target loss function.

[0061] Furthermore, in some embodiments, S122 includes the following steps:

[0062] Determining a first contrast loss function based on the text-image similarity between the job text feature vector and the job image feature vector in each of the positive sample pairs, and the text-image similarity between the job text feature vector and the job image feature vector in each of the negative sample pairs;

[0063] Determining a second contrast loss function based on the image-text similarity between the task image feature vector and the task text feature vector in each of the positive sample pairs;

[0064] The target loss function is determined based on the first contrast loss function and the second contrast loss function.

[0065] Specifically, the cross-modal semantic alignment model is trained by minimizing the multi-positive contrast loss function. This minimizes the loss function of the cross-modal semantic alignment model, ensuring that matching image-text vectors are closer in the feature space and mismatched image-text vectors are farther apart. This ultimately maps text and images into a unified semantic space. The multi-positive contrast loss function is an extension of the standard contrast loss, allowing a single text to be paired with multiple images simultaneously as positive samples. This improves alignment robustness by aggregating the similarities of multiple positive samples.

[0066] Among them, the formula of the multi-positive sample contrast loss function is as follows:

[0067] Text-oriented contrast loss function L text :

[0068]

[0069] Where B represents the number of job images and job texts involved in the calculation during training; i represents the index number of the job text; P(i) represents the set of all image indexes corresponding to job text i; t i Represents the feature vector of the i-th job text; i j Represents the feature vector of the j-th job image; i k represents the feature vector of the k-th job image; τ represents the temperature parameter; s(.) represents the cosine similarity function; j∈P(i) represents the positive sample; represents negative samples;

[0070] Contrast loss function L for image direction image :

[0071]

[0072] Where B represents the number of job images and job texts involved in the calculation during training; j represents the index number of the job image; i j Represents the feature vector of the jth job image; t j The feature vector representing the jth job text; t k represents the feature vector of the k-th job text; s(.) is the cosine similarity function; k represents the index number of the job text; τ represents the temperature parameter;

[0073] Total loss function L:

[0074]

[0075] Among them, L text Represents the contrast loss function of text direction, L image Contrastive loss function representing image orientation.

[0076] Traditional maintenance systems often process images and text separately, and their models can only learn the features of a single modality. Multimodal alignment models can simultaneously understand image structure and language semantics, resulting in stronger semantic generalization capabilities. Multimodal alignment models can better handle the complexities of power equipment scenarios and the non-standard language descriptions found in real-world applications.

[0077] Specifically, in another embodiment, S120 includes the following steps:

[0078] Performing image feature extraction on the job image in the positive sample pair according to a pre-trained image encoder to obtain a target image vector;

[0079] Performing text feature extraction on the job text in the positive sample pair according to a pre-trained text encoder to obtain a target text vector;

[0080] The target image vector and the target text vector are input into the cross-modal semantic alignment model to obtain a multimodal operation knowledge vector.

[0081] The image encoder is a pre-trained model that segments the input image into fixed-size pixel blocks and extracts global and local features through a self-attention mechanism. The text encoder is a pre-trained language model that performs word segmentation and position encoding on the input text and uses a self-attention mechanism to understand the contextual semantics.

[0082] Specifically, for the job image, the target image vector is generated by the image encoder. Its input resolution can be adjusted to 224×224, and the image features are extracted through the Vision Transformer-Base / 16, ViT-B / 16, and the output is a 768-dimensional vector. By extracting features from the target image data through the pre-trained image encoder, high-level semantic information of the image, such as device type, fault area, environmental characteristics, etc., can be quickly extracted without large-scale annotation, thereby converting the original image into a structured semantic representation that can be understood by the machine. This not only facilitates efficient processing and similarity retrieval of subsequent models, but also significantly reduces computing resource consumption and improves system response speed.

[0083] The target text vector generated by the text encoder for the job text can be achieved through a pre-trained Chinese BERT model. The Chinese BERT model generates a 768-dimensional vector representation (also known as the target text vector) that represents the overall semantics based on the contextual relationships, word dependencies, and other information of the job text. The pre-trained language knowledge of the Chinese BERT model can understand the contextual meaning of terms in the power field (such as the difference in "discharge" in different equipment scenarios), thereby reducing the semantic ambiguity of professional texts. By extracting features from the job text through a pre-trained text encoder, key information such as the equipment name and event impact can be accurately captured, resulting in a text vector representation with semantic depth.

[0084] The target image vector and target text vector are fed into a pre-trained cross-modal semantic alignment model. This model projects the two vectors into the same vector space and outputs a fused multimodal semantic vector, also known as a "multimodal inspection knowledge vector," which combines both the visual information of the image and the linguistic information of the text. This cross-modal vector space supports cross-verification between image and text, effectively avoiding errors caused by manual annotation.

[0085] In addition, the multimodal inspection knowledge base can be Milvus Vector Database (MVD), which is an open source vector database system designed specifically for storing and retrieving large-scale high-dimensional vectors. It supports high-performance similarity retrieval and improves vector retrieval speed and accuracy.

[0086] Specifically, the target image vector and target text vector, processed by the image encoder and text encoder, are first written into the vector database system Milvus and structurally annotated. The target image vector and target text vector serve as image-text vector samples, which are then encoded with metadata based on the device type and fault level of the device to which they belong. The device type is represented using a hierarchical structure of "voltage level + device category + device model." Fault levels are classified according to the "Q / GDW 519-2010 Distribution Network Operation Regulations" standard using one-hot vector encoding, and are categorized into three categories: "general," "serious," and "critical."

[0087] Then, based on the combination of device type and fault level, a joint hash table index structure is constructed to serve as the basis for vector bucketing. Each hash bucket corresponds to a specific "device type-fault level" combination. The system then divides all image and text vector samples into their corresponding hash buckets, achieving multi-dimensional semantic-level partitioning management.

[0088] Furthermore, an independent, improved Hierarchical Navigable Small World (HNSW) graph index structure is constructed within each hash bucket. This structure supports fast nearest neighbor searches for nodes in the graph and allows for parameter optimization (such as the number of search levels and links) based on the characteristics of each bucket. This index structure enables efficient coarse screening of vectors within the bucket.

[0089] Building on the aforementioned primary indexing, we further introduce a product quantization (PQ) indexing mechanism to compress high-dimensional vectors within each hash bucket. PQ decomposes the original vector into multiple sub-vectors and establishes a codebook for each sub-vector. This enables efficient storage and compressed representation of high-dimensional vectors, reducing memory usage.

[0090] When an inspection robot or client initiates a query request, the system prioritizes locating the target subbucket using the device-fault joint hash table based on the device type and fault level information included in the request. Within the target subbucket, a hierarchical vector nearest neighbor search is performed using the HNSW graph to quickly obtain the TopK candidate vectors. Based on this, the PQ mechanism is used to decode and compensate for the candidate vectors, restoring the compressed vectors to an approximate original vector. Precise distance calculation and sorting are then performed on the candidate results to obtain the final refined ranking.

[0091] If the query request fails to accurately match a specific sub-bucket, the system degenerates into a coarse screening and fine sorting process within the global vector range, ensuring the robustness and recall capability of the system when the image and text vector sample labels are missing or ambiguous.

[0092] Through the above steps, this embodiment significantly improves the retrieval speed and accuracy of multimodal operation and inspection knowledge vectors while ensuring the controllable scale of vector storage, and meets the real-time semantic query needs of power operation robots in inspection, diagnosis, decision-making and other links.

[0093] S130 , in response to the operation request text for the operation and inspection robot, semantically aligning the operation request text based on the cross-modal semantic alignment model to obtain a semantically aligned operation request text vector.

[0094] First, the system receives a job request text input from a user or job scheduling system. This job request text can be a natural language description and includes, but is not limited to, the target device name, job type, task action, and a description of the work environment. For example, the text might be, "Please perform live infrared temperature measurement on the main switch contacts of a 10kV distribution cabinet."

[0095] The system then feeds the job request text into a trained text encoder module. This encoder is part of the cross-modal semantic alignment model, which has been pre-trained to model the language characteristics of the power operation and maintenance domain using positive sample image-text pairs. This encoder performs semantic modeling on the job request text, extracting implicit semantic information such as key entities (e.g., "10kV distribution cabinet," "main switch contactor"), action verbs (e.g., "temperature measurement"), and job constraints (e.g., "energized"), and converts this information into a fixed-length vector representation of the job request text.

[0096] To achieve cross-modal retrieval and alignment, the job requests that the text vector be mapped to a shared vector space consistent with the image vector, namely the aforementioned cross-modal semantic space, so that it can be compared for similarity with the multimodal knowledge vectors stored in the knowledge base.

[0097] S140 , searching the multimodal operation and inspection knowledge base for a target knowledge vector that is similar to the operation request text vector.

[0098] Specifically, in some embodiments, Figure 3 As shown, S140 includes the following steps:

[0099] S141: Determine a query vector and query keywords based on the job request text.

[0100] Specifically, the system receives a query request (job request text) input by the user. The query request may include image data, text data, or any combination thereof. For image queries, the system uses an image encoder consistent with the training phase to extract features and obtain an image query vector. For text queries, the system uses a pre-trained text encoder to extract semantic features and obtain a text query vector. It then further performs keyword extraction on the text to extract core semantic keywords and form a set of query keywords. The image query vector and the text query vector are used together as query vectors. By standardizing the input query request into a structured representation, including one or more high-dimensional query vectors and a set of keywords, the subsequent retrieval consistency and semantic integrity are ensured.

[0101] S142 : Based on the query vector, searching the multimodal operation and inspection knowledge base for a plurality of first operation text knowledge vectors and a plurality of first operation image knowledge vectors whose vector similarity with the query vector meets a first similarity condition.

[0102] Among them, the first similarity condition refers to the similarity threshold condition between the query vector and the target knowledge vector (including the job text knowledge vector and the job image knowledge vector) in the multimodal operation and inspection knowledge base (the similarity threshold condition is set to the minimum distance between the vectors, for example).

[0103] Specifically, the query vector is submitted to the multimodal inspection knowledge base for similarity vector matching. The system then uses a predefined similarity metric (e.g., cosine similarity or inner product) to retrieve the top-K first-job text knowledge vectors and first-job image knowledge vectors with the smallest distance to the query vector. The system quickly selects the K first-job text knowledge vectors and first-job image knowledge vectors that are closest to the query in the semantic space as potentially relevant knowledge content, providing a candidate range for subsequent comprehensive ranking.

[0104] S143, based on the query keyword, retrieve multiple first original work texts whose keyword similarity with the query keyword meets the second similarity condition in the original work texts corresponding to each work text knowledge vector in the multimodal operation and inspection knowledge base to determine the corresponding multiple second work text knowledge vectors.

[0105] Among them, the second similarity condition refers to calculating the keyword matching between the query keyword and the original data of the job text, and considering that the keyword matching score is greater than the preset score threshold as meeting the similarity condition.

[0106] Specifically, based on the extracted query keywords, the Best Matching 25 (BM25) text matching algorithm is used to calculate the keyword matching degree of the original homework text corresponding to each homework text knowledge vector, and output the keyword matching score. According to the keyword matching score, multiple first original homework texts whose retrieval keyword similarity is greater than a preset score threshold are obtained to determine the corresponding multiple second homework text knowledge vectors. As an improved TF-IDF ranking algorithm, BM25 can effectively measure the keyword coverage and relevance between short texts. On the basis of vector semantic matching, explicit semantic supplementation at the keyword level is further introduced to improve the accuracy of text retrieval and the interpretability of the results.

[0107] S144 : Determine a target knowledge vector based on the multiple first job text knowledge vectors, the multiple first job image knowledge vectors, and the multiple second job text knowledge vectors.

[0108] Furthermore, in some embodiments, S144 includes the following steps:

[0109] determining a comprehensive score for each of the first job image knowledge vectors based on a vector similarity between each of the first job image knowledge vectors and the query vector;

[0110] performing a union process on the plurality of first homework text knowledge vectors and the plurality of second homework text knowledge vectors to obtain a third homework text knowledge vector set;

[0111] Determining a comprehensive score for each of the third homework text knowledge vectors based on a vector similarity between each of the third homework text knowledge vectors in the third homework text knowledge vector set and the query vector, and a keyword similarity between each of the third homework text knowledge vectors and the query keyword;

[0112] Based on the comprehensive scores of the third job text knowledge vectors and the comprehensive scores of the first job image knowledge vectors, a target knowledge vector is determined among the third job text knowledge vectors and the first job image knowledge vectors.

[0113] Specifically, the vector similarity can be calculated using metrics such as cosine similarity or inner product, and the comprehensive score of each first job image knowledge vector can be obtained by weighted summing the vector similarities between each first job image knowledge vector and the query vector.

[0114] The first homework text knowledge vector set is combined with the second homework text knowledge vector set to form a third homework text knowledge vector set. Text vectors from different sources can be combined to enrich the homework text knowledge vector samples.

[0115] Calculate the vector similarity between each third assignment text knowledge vector and the query vector, and calculate the keyword similarity between each third assignment text knowledge vector and the keyword vector. Based on the calculated vector similarity and keyword similarity, perform a weighted sum of the two according to a preset comprehensive score calculation rule to obtain a comprehensive score for each third assignment text knowledge vector. The preset comprehensive score calculation rule is: comprehensive score = vector similarity * 80% + keyword similarity * 20%.

[0116] The system sorts candidate vectors (composed of the knowledge vectors for each third-party task text and each task image) in descending order based on their comprehensive scores. It then selects the top-N highest-scoring vectors as the final target knowledge vectors, outputs them, and returns them to the user or task robot. The final retrieval results, combining the dual metrics of text-image semantic fusion and specialized semantic coverage, are more robust and reliable, and can be widely applied in scenarios such as equipment failure case backtracking, historical processing suggestion extraction, and knowledge graph node recall in maintenance robot operations.

[0117] In summary, this embodiment effectively improves the efficiency and accuracy of knowledge retrieval in complex operation and maintenance scenarios by constructing a multimodal query system that combines images and texts, combining vector semantic matching with keyword matching algorithms, and unifying the sorting through a comprehensive scoring mechanism. It is particularly suitable for industrial operation and maintenance scenarios with heterogeneous image and text input and strong expression uncertainty.

[0118] S150 , generating an operation action instruction of the operation and inspection robot based on the target knowledge vector and the operation request text.

[0119] Specifically, in some embodiments, S150 includes the following steps:

[0120] Initially learning the target knowledge vector and the job request text using a preset lightweight model to obtain a current job query vector;

[0121] If a first similarity between the operation process vector in the current operation query vector and a first standard operation process vector in a preset standard operation process library is greater than a preset first similarity threshold, and a second similarity between the operation environment vector in the current operation query vector and a standard operation environment vector corresponding to the first standard operation process vector is greater than a preset second similarity threshold, determining an operation action instruction of the operation and inspection robot based on the operation instruction corresponding to the first standard operation process vector;

[0122] When the first similarity is less than the first similarity threshold, and / or the second similarity is less than the second similarity threshold, the current operation query vector is input into a preset deep reasoning model to obtain an optimized operation instruction, and the optimized operation instruction is used as the operation action instruction of the operation and inspection robot.

[0123] Specifically, the preset lightweight model can be the TinyLLAMA-1.1B lightweight model. The system uses the target knowledge vector obtained in the previous step as a priori reference, combines it with the current query request, and performs feature fusion using the preset lightweight fusion model to output a unified high-dimensional job query vector, which is recorded as the current job query vector.

[0124] Calculate the first similarity between the operation process vector in the current operation query vector and the first standard operation process vector in the preset standard operation process library; and calculate the second similarity between the operation environment vector in the current operation query vector and the standard operation environment vector corresponding to the first standard operation process vector. Among them, the similarity calculation can adopt a measurement method such as cosine similarity or inner product. When the first similarity is greater than the preset first similarity threshold and the second similarity is greater than the preset second similarity threshold, the operation instruction corresponding to the first standard operation process vector is determined as the operation action instruction of the operation and inspection robot. That is, when the current operation process and the current operation environment are highly matched with the operation process and the operation environment in the preset standard operation process library respectively, the system can directly reuse the experience template to improve the response speed and operation reliability.

[0125] The preset deep reasoning model can be a deep attention network (such as deepseek-671B), a graph neural network or other semantic generation models. Its function is to perform dynamic reasoning and optimization based on the current context information and prior knowledge, and output an optimized operation instruction that meets the current needs as the final operation action instruction.

[0126] To sum up, the job instruction generation method provided in this embodiment takes into account both standardized process reuse and complex scenario reasoning capabilities. It combines multimodal semantic vector representation, lightweight fusion model and deep reasoning model to realize full-link automated decision-making from query intent to action execution instructions. It is particularly suitable for real-time job guidance scenarios such as power inspection, remote control, and complex maintenance.

[0127] Furthermore, in some embodiments, after step S150, the method further includes the following steps:

[0128] When the operation action instruction includes an illegal operation, determining the risk of the operation action instruction causing an accident;

[0129] When the risk is less than a first threshold, replacing the illegal operation with a preset safety action template;

[0130] When the risk is greater than the first threshold, the operation action instruction is adjusted based on a preset deep reasoning model to generate an adjusted operation action instruction.

[0131] Specifically, international standards are preset to build a structured safety rule base, and various safety specifications, operating standards and violation behavior characteristics are pre-stored in the safety rule base. Real-time verification is used to generate operating action instructions using rule matching. The judgment of illegal operations is based on multiple key factors, including whether the operation and maintenance robot violates the safety distance requirements, whether it performs actions that exceed the load or movement limits, whether it uses the wrong tools or operating modes, whether it contains unauthorized or abnormal operation steps, and whether it continues to operate in the case of abnormal environmental monitoring. For example, if the operation and maintenance robot attempts to directly grab a live wire with its robotic arm, or performs high-altitude precision operations when the wind speed exceeds the standard, these are all illegal operations.

[0132] The system matches and verifies the action information in the operation action instruction with the action information in the preset safety rule library. If it is found that the operation action instruction contains illegal operations, the system determines the risk of the operation action instruction causing an accident and triggers the secondary rollback mechanism. The secondary rollback mechanism includes primary correction and deep correction. The trigger condition for the primary correction is a minor violation, that is, although the operation poses a safety hazard, it will not directly lead to serious consequences. When the risk probability of the operation action instruction causing an accident is less than the first threshold, the system will replace the illegal operation in the operation action instruction with a predefined safety action template. Among them, the first threshold is set according to actual needs. For example, assuming the first threshold is 60%, if the robot plans to continue working when the wind speed exceeds the set threshold (for example, 10 meters / second), the predicted risk probability of an accident is 40%. The system will adjust to "delay operation and wait for the wind speed to return to normal" to reduce the risk.

[0133] Deep corrections are triggered by serious violations, meaning they could cause equipment damage, threaten personal safety, or even trigger a power system accident. If the risk of an accident caused by an action instruction exceeds a first threshold, the system invokes the deep inference layer to regenerate a new set of action instructions that comply with safety regulations. For example, if the first threshold is 60%, and the robot detects excessive humidity but still plans to perform high-risk live work, the predicted risk of an accident is 85%. The system may reschedule the operation or provide alternative insulation protection solutions to ensure safe operation.

[0134] In summary, this embodiment further enhances the practicality and robustness of the operation action instruction generation system by introducing a processing mechanism that combines violation detection, hierarchical response mechanism and deep reasoning optimization, so that the operation and maintenance robot can not only "take action" but also have the ability to "judge right and wrong and correct instructions", truly realizing the intelligence, standardization and full-process risk control of power operations.

[0135] Figure 4 It is a structural block diagram of an operation decision-making device of an inspection robot according to an embodiment of the present invention.

[0136] like Figure 4 As shown, the operation decision-making device of the operation and inspection robot may include:

[0137] A construction module 510 is used to construct a plurality of positive sample pairs and a plurality of negative sample pairs based on the historical operation request texts and historical operation images of the operation and inspection robot;

[0138] A training module 520 is configured to train a cross-modal semantic alignment model based on the multiple positive sample pairs and the multiple negative sample pairs, perform semantic alignment on each of the positive sample pairs using the trained cross-modal semantic alignment model, obtain a semantically aligned knowledge vector, and store the knowledge vector in a multimodal inspection knowledge base;

[0139] a semantic alignment module 530 for performing semantic alignment on a job request text for the operation and inspection robot based on the cross-modal semantic alignment model in response to the job request text to obtain a semantically aligned job request text vector;

[0140] A retrieval module 540 is configured to retrieve a target knowledge vector similar to the job request text vector in the multimodal operation and inspection knowledge base;

[0141] The generation module 550 is used to generate the operation action instructions of the operation and inspection robot based on the target knowledge vector and the operation request text.

[0142] In some embodiments, as Figure 5As shown, the training module 520 includes:

[0143] A feature vector obtaining unit 521 is configured to input the plurality of positive sample pairs and the plurality of negative sample pairs into the cross-modal semantic alignment model, respectively, to obtain a feature vector of each semantically aligned positive sample pair and a feature vector of each negative sample pair, wherein the positive sample pair includes a sample pair consisting of a matching job text and one or more job images, and the negative sample pair includes a sample pair consisting of a mismatched job text and one or more job images;

[0144] a target loss function determining unit 522, configured to determine a target loss function based on the feature vectors of each of the positive sample pairs and the feature vectors of each of the negative sample pairs;

[0145] The model parameter adjustment unit 523 is used to adjust the model parameters of the cross-modal semantic alignment model based on the target loss function.

[0146] In some embodiments, as Figure 6 As shown, the target loss function determining unit 522 includes:

[0147] A first contrast loss function determination subunit 5221 is configured to determine a first contrast loss function based on the text-image similarity between the job text feature vector and the job image feature vector in each of the positive sample pairs, and the text-image similarity between the job text feature vector and the job image feature vector in each of the negative sample pairs;

[0148] A second contrast loss function determination subunit 5222 is configured to determine a second contrast loss function based on the image-text similarity between the job image feature vector and the job text feature vector in each of the positive sample pairs;

[0149] The target loss function determination subunit 5223 is used to determine the target loss function based on the first contrast loss function and the second contrast loss function.

[0150] In some embodiments, as Figure 7 As shown, the retrieval module 540 includes:

[0151] A first determining unit 541 is configured to determine a query vector and a query keyword based on the job request text;

[0152] A retrieval unit 542 is configured to retrieve, based on the query vector, a plurality of first operation text knowledge vectors and a plurality of first operation image knowledge vectors whose vector similarity with the query vector meets a first similarity condition in the multimodal operation and inspection knowledge base;

[0153] A second determining unit 543 is configured to retrieve, based on the query keyword, a plurality of first original work texts corresponding to the respective work text knowledge vectors in the multimodal operation and inspection knowledge base, whose keyword similarity with the query keyword meets a second similarity condition, to determine a plurality of corresponding second work text knowledge vectors;

[0154] The third determining unit 544 is configured to determine a target knowledge vector based on the plurality of first job text knowledge vectors, the plurality of first job image knowledge vectors, and the plurality of second job text knowledge vectors.

[0155] In some embodiments, as Figure 8 As shown, the third determining unit 544 includes:

[0156] A first comprehensive score determination subunit 5441 is configured to determine a comprehensive score for each of the first job image knowledge vectors based on a vector similarity between the first job image knowledge vector and the query vector;

[0157] a union processing subunit 5442, configured to perform a union process on the plurality of first homework text knowledge vectors and the plurality of second homework text knowledge vectors to obtain a third homework text knowledge vector set;

[0158] A second comprehensive score determination subunit 5443 is configured to determine a comprehensive score for each of the third assignment text knowledge vectors based on a vector similarity between each of the third assignment text knowledge vectors in the third assignment text knowledge vector set and the query vector, and a keyword similarity between each of the third assignment text knowledge vectors and the query keyword.

[0159] The target knowledge vector determination subunit 5444 is used to determine the target knowledge vector in each of the third job text knowledge vectors and each of the first job image knowledge vectors based on the comprehensive score of each of the third job text knowledge vectors and the comprehensive score of each of the first job image knowledge vectors.

[0160] In some embodiments, as Figure 9 As shown, the generating module 550 includes:

[0161] An initial learning unit 551 is configured to perform initial learning on the target knowledge vector and the job request text using a preset lightweight model to obtain a current job query vector;

[0162] a fourth determining unit 552 configured to determine, based on the operation instruction corresponding to the first standard operation process vector, an operation action instruction for the operation and inspection robot when a first similarity between the operation process vector in the current operation query vector and a first standard operation process vector in a preset standard operation process library is greater than a preset first similarity threshold, and a second similarity between the operation environment vector in the current operation query vector and a standard operation environment vector corresponding to the first standard operation process vector is greater than a preset second similarity threshold;

[0163] The optimization unit 553 is used to input the current operation query vector into a preset deep reasoning model to obtain an optimized operation instruction when the first similarity is less than the first similarity threshold and / or the second similarity is less than the second similarity threshold, and use the optimized operation instruction as the operation action instruction of the operation and inspection robot.

[0164] In some embodiments, as Figure 10 As shown, after the generating module 550, the apparatus further includes:

[0165] A risk determination module 560 is configured to determine the risk of an accident caused by the operation instruction when the operation instruction contains an illegal operation;

[0166] A replacement module 570, configured to replace the illegal operation with a preset security action template when the risk is less than a first threshold;

[0167] The adjustment module 580 is used to adjust the operation action instruction based on a preset deep reasoning model when the risk is greater than the first threshold, and generate an adjusted operation action instruction.

[0168] For the description of specific functions and examples of each module and submodule of the system in the embodiment of the present invention, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0169] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0170] An embodiment of the present invention further provides an operation decision system for an inspection robot, comprising:

[0171] at least one processor; and

[0172] a memory communicatively connected to the at least one processor; wherein,

[0173] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods in the embodiments of the present invention.

[0174] The beneficial effects of the operation decision-making system of the operation and inspection robot according to the embodiment of the present invention are equivalent to the beneficial effects of the operation decision-making method of the operation and inspection robot described above, and are not described in detail here.

[0175] An embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute any one of the methods in the embodiments of the present invention.

[0176] The beneficial effects of the storage medium of the present invention are equivalent to the beneficial effects of the operation decision-making method of the above-mentioned operation and inspection robot, and will not be described in detail here.

[0177] Figure 11 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present invention is shown. Electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 800 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0178] like Figure 11 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0179] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0180] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the operation decision-making method of the inspection robot. For example, in some embodiments, the operation decision-making method of the inspection robot can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the operation decision-making method of the inspection robot described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the operation decision-making method of the operation and inspection robot in any other appropriate manner (for example, by means of firmware).

[0181] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0183] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0184] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0185] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0187] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0188] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for decision-making on operation of an inspection robot, characterized in that: include: Based on the historical job request text and historical job images of the operation and inspection robot, multiple positive sample pairs and multiple negative sample pairs are constructed; Based on the multiple positive sample pairs and the multiple negative sample pairs, a cross-modal semantic alignment model is trained, and each of the positive sample pairs is semantically aligned using the trained cross-modal semantic alignment model to obtain a semantically aligned knowledge vector, and the knowledge vector is stored in a multimodal operation and inspection knowledge base; In response to a job request text for the operation and inspection robot, semantically aligning the job request text based on the cross-modal semantic alignment model to obtain a semantically aligned job request text vector; Retrieving a target knowledge vector similar to the job request text vector in the multimodal operation and inspection knowledge base; Based on the target knowledge vector and the operation request text, an operation action instruction of the operation and inspection robot is generated.

2. The method according to claim 1, characterized in that The training of the cross-modal semantic alignment model based on the multiple positive sample pairs and the multiple negative sample pairs includes: Inputting the multiple positive sample pairs and the multiple negative sample pairs into the cross-modal semantic alignment model respectively to obtain semantically aligned feature vectors of each positive sample pair and each negative sample pair, wherein the positive sample pair includes a sample pair consisting of a job text and one or more job images that match each other, and the negative sample pair includes a sample pair consisting of a job text and one or more job images that do not match each other; Determining a target loss function based on the feature vectors of each of the positive sample pairs and the feature vectors of each of the negative sample pairs; Based on the target loss function, model parameters of the cross-modal semantic alignment model are adjusted.

3. The method according to claim 2, characterized in that The determining of a target loss function based on the feature vectors of each of the positive sample pairs and the feature vectors of each of the negative sample pairs includes: Determining a first contrast loss function based on the text-image similarity between the job text feature vector and the job image feature vector in each of the positive sample pairs, and the text-image similarity between the job text feature vector and the job image feature vector in each of the negative sample pairs; Determining a second contrast loss function based on the image-text similarity between the task image feature vector and the task text feature vector in each of the positive sample pairs; The target loss function is determined based on the first contrast loss function and the second contrast loss function.

4. The method according to claim 1, wherein Retrieving a target knowledge vector similar to the job request text vector in the multimodal operation and inspection knowledge base includes: Determining a query vector and a query keyword based on the job request text; Based on the query vector, searching the multimodal operation and inspection knowledge base for a plurality of first operation text knowledge vectors and a plurality of first operation image knowledge vectors whose vector similarity with the query vector meets a first similarity condition; Based on the query keyword, searching, from the original operation texts corresponding to the respective operation text knowledge vectors in the multimodal operation and inspection knowledge base, a plurality of first original operation texts whose keyword similarity with the query keyword meets a second similarity condition, to determine a plurality of corresponding second operation text knowledge vectors; A target knowledge vector is determined based on the plurality of first job text knowledge vectors, the plurality of first job image knowledge vectors, and the plurality of second job text knowledge vectors.

5. The method according to claim 4, characterized in that The determining of a target knowledge vector based on the plurality of first job text knowledge vectors, the plurality of first job image knowledge vectors, and the plurality of second job text knowledge vectors includes: determining a comprehensive score for each of the first job image knowledge vectors based on a vector similarity between each of the first job image knowledge vectors and the query vector; performing a union process on the plurality of first homework text knowledge vectors and the plurality of second homework text knowledge vectors to obtain a third homework text knowledge vector set; Determining a comprehensive score for each of the third assignment text knowledge vectors based on a vector similarity between each of the third assignment text knowledge vectors in the third assignment text knowledge vector set and the query vector, and a keyword similarity between each of the third assignment text knowledge vectors and the query keyword; Based on the comprehensive scores of the third job text knowledge vectors and the comprehensive scores of the first job image knowledge vectors, a target knowledge vector is determined among the third job text knowledge vectors and the first job image knowledge vectors.

6. The method according to claim 1, characterized in that The step of generating an operation action instruction for the operation and inspection robot based on the target knowledge vector and the operation request text includes: Initially learning the target knowledge vector and the job request text using a preset lightweight model to obtain a current job query vector; If a first similarity between the operation process vector in the current operation query vector and a first standard operation process vector in a preset standard operation process library is greater than a preset first similarity threshold, and a second similarity between the operation environment vector in the current operation query vector and a standard operation environment vector corresponding to the first standard operation process vector is greater than a preset second similarity threshold, determining an operation action instruction of the operation and inspection robot based on the operation instruction corresponding to the first standard operation process vector; When the first similarity is less than the first similarity threshold, and / or the second similarity is less than the second similarity threshold, the current operation query vector is input into a preset deep reasoning model to obtain an optimized operation instruction, and the optimized operation instruction is used as the operation action instruction of the operation and inspection robot.

7. The method according to claim 1, characterized in that After generating the operation action instruction of the operation inspection robot based on the target knowledge vector and the operation request text, the method further includes: When the operation action instruction includes an illegal operation, determining the risk of the operation action instruction causing an accident; When the risk is less than a first threshold, replacing the illegal operation with a preset safety action template; When the risk is greater than the first threshold, the operation action instruction is adjusted based on a preset deep reasoning model to generate an adjusted operation action instruction.

8. An operation decision-making device for an inspection robot, characterized in that: include: A construction module is used to construct multiple positive sample pairs and multiple negative sample pairs based on the historical job request text and historical job images of the operation and inspection robot; A training module is configured to train a cross-modal semantic alignment model based on the multiple positive sample pairs and the multiple negative sample pairs, and perform semantic alignment on each of the positive sample pairs using the trained cross-modal semantic alignment model to obtain a semantically aligned knowledge vector, and store the knowledge vector in a multimodal operation and inspection knowledge base; a semantic alignment module, configured to respond to a job request text for the operation and inspection robot and perform semantic alignment on the job request text based on the cross-modal semantic alignment model to obtain a semantically aligned job request text vector; A retrieval module, configured to retrieve a target knowledge vector similar to the job request text vector in the multimodal operation and inspection knowledge base; A generation module is used to generate an operation action instruction of the operation and inspection robot based on the target knowledge vector and the operation request text.

9. The device according to claim 8, characterized in that The retrieval module includes: A first determining unit, configured to determine a query vector and a query keyword based on the job request text; a retrieval unit configured to retrieve, based on the query vector, a plurality of first operation text knowledge vectors and a plurality of first operation image knowledge vectors whose vector similarity with the query vector meets a first similarity condition in the multimodal operation and inspection knowledge base; A second determining unit is configured to retrieve, based on the query keyword, a plurality of first original work texts corresponding to the respective work text knowledge vectors in the multimodal operation and inspection knowledge base, whose keyword similarity with the query keyword meets a second similarity condition, to determine a plurality of corresponding second work text knowledge vectors; The third determining unit is configured to determine a target knowledge vector based on the plurality of first job text knowledge vectors, the plurality of first job image knowledge vectors, and the plurality of second job text knowledge vectors.

10. The device according to claim 8, characterized in that The generation module includes: An initial learning unit, configured to perform initial learning on the target knowledge vector and the job request text using a preset lightweight model to obtain a current job query vector; a fourth determining unit configured to determine, based on the operation instruction corresponding to the first standard operating procedure vector, an operation action instruction for the operation and inspection robot when a first similarity between the operation process vector in the current operation query vector and a first standard operating process vector in a preset standard operating process library is greater than a preset first similarity threshold, and a second similarity between the operation environment vector in the current operation query vector and a standard operating environment vector corresponding to the first standard operating process vector is greater than a preset second similarity threshold; An optimization unit is used to input the current operation query vector into a preset deep reasoning model to obtain an optimized operation instruction when the first similarity is less than the first similarity threshold and / or the second similarity is less than the second similarity threshold, and use the optimized operation instruction as the operation action instruction of the operation and inspection robot.

11. The device according to claim 8, characterized in that After the generation module, the device further includes: A risk determination module, configured to determine the risk of an accident caused by the operation action instruction when the operation action instruction contains an illegal operation; a replacement module, configured to replace the illegal operation with a preset security action template when the risk is less than a first threshold; An adjustment module is used to adjust the operation action instruction based on a preset deep reasoning model when the risk is greater than the first threshold, and generate an adjusted operation action instruction.

12. An operation decision system for an inspection robot, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images