Visual voice navigation method, system and equipment for transformer substation and medium

Abstract

The invention discloses a visual voice navigation method, system and equipment for a transformer substation and a medium. The method comprises the following steps: acquiring visual information and a voice instruction of a current environment; and inputting the visual information of the current environment and the voice instruction into the trained multi-mode visual voice navigation model to obtain a robot executable instruction, and controlling the robot according to the robot executable instruction until the robot reaches a target position. The multi-mode visual voice navigation model comprises an auditory encoder, a visual encoder, a multiprocessor module, a decoder, a knowledge graph module and an output layer, the auditory encoder and the visual encoder are connected with the multiprocessor module, the multiprocessor module is connected with the decoder, the decoder and the knowledge graph module are connected with the output layer, and the output layer is connected with the decoder. The method, the system, the equipment and the medium can improve the accuracy and the stability of navigation.

Description

Technical Field

[0001] The invention belongs to the technical field of visual voice navigation, and relates to a method, system, equipment and medium for visual voice navigation of a substation. Background Art

[0002] With the rapid development of the power system, substations are the core nodes of the power grid, and their safety and reliability are of vital importance. Traditional substation maintenance work relies on manual labor, which has problems such as low efficiency and poor safety. In order to improve inspection efficiency, substation inspection robots have been widely used in recent years. During the maintenance of power equipment, robots need to be able to accurately navigate and locate. However, due to the wide variety and different shapes of power equipment, as well as the complex and changeable working environment, the existing robot navigation and positioning technology often cannot meet actual needs.

[0003] At present, the navigation methods commonly used by substation inspection robots mainly include visual navigation, laser navigation, inertial navigation, etc. Visual navigation is to capture images of the surrounding environment through a camera, and use image processing and recognition technology to identify landmarks and obstacles in the environment, thereby realizing the robot's navigation and obstacle avoidance. The existing visual navigation methods also have the following shortcomings: when faced with complex environments, they are often unable to accurately identify and avoid obstacles, which may lead to collisions and misjudgments during the inspection process; they cannot accurately recognize their own position in the substation in real time and dynamically, and cannot be based on the site to carry out actual and effective path planning and adjustment, affecting the flexibility, initiative and efficiency of the inspection. The reasons for these problems are as follows:

[0004] Single information: Navigation solutions that rely on single sensor information cannot fully utilize multimodal information in the environment, resulting in limited navigation accuracy and adaptability.

[0005] Insufficient fusion mechanism: Even though some methods try to fuse information from multiple sensors, the lack of an efficient fusion mechanism results in poor information fusion and cannot significantly improve navigation performance. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a substation visual voice navigation method, system, device and medium, which can improve the accuracy and stability of navigation.

[0007] To achieve the above object, the present invention discloses a substation visual voice navigation method, comprising:

[0008] Get visual information of the current environment and voice commands;

[0009] The visual information of the current environment and the voice commands are input into the trained multimodal visual voice navigation model to obtain executable commands for the robot, and the robot is controlled according to the executable commands until the target position is reached. The multimodal visual voice navigation model includes an auditory encoder, a visual encoder, a multiprocessor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multiprocessor module, the multiprocessor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer.

[0010] The further improvement of the substation visual voice navigation method of the present invention is:

[0011] Furthermore, before inputting the visual information and voice command of the current environment into the trained multimodal visual voice navigation model, the method further includes:

[0012] Build a diverse dataset containing visual images of substation environments and corresponding voice commands;

[0013] Construct a multimodal visual and speech navigation model;

[0014] The multimodal visual-speech navigation model is trained based on the diversified data set to obtain a trained multimodal visual-speech navigation model.

[0015] Furthermore, the loss function in the process of training the multimodal visual speech navigation model based on the diversified data set is:

[0016] The loss function during training is:

[0017] L=λL int +(1-λ)(L nav +γL kg )L (5)

[0018] Among them, L int is the interaction loss, which is used to reflect the interaction between the robot and the environment when performing tasks; L nav is the navigation loss, which is used to measure the accuracy or efficiency of the robot’s navigation strategy. kg is a loss term related to the knowledge graph, which is used to evaluate the accuracy or consistency of the robot when using the knowledge graph for navigation. λ and γ are parameters used to adjust the balance loss.

[0019] Furthermore, the visual encoder outputs RGB features F (RGB) , semantic segmentation feature F (s) , deep features F (d) and region proposal feature F (0) .

[0020] The invention discloses a substation visual voice navigation system, comprising:

[0021] The acquisition module is used to obtain visual information and voice commands of the current environment;

[0022] The navigation module is used to input the visual information of the current environment and the voice instructions into the trained multimodal visual voice navigation model to obtain the robot executable instructions, and control the robot according to the robot executable instructions until it reaches the target position. The multimodal visual voice navigation model includes an auditory encoder, a visual encoder, a multi-processor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multi-processor module, the multi-processor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer.

[0023] The further improvement of the substation visual voice navigation system of the present invention is:

[0024] Furthermore, it also includes:

[0025] A first building module is used to build a diverse dataset containing visual images of the substation environment and corresponding voice commands;

[0026] The second building module is used to build a multimodal visual speech navigation model;

[0027] A training module is used to train the multimodal visual-speech navigation model based on the diversified data set to obtain a trained multimodal visual-speech navigation model.

[0028] Furthermore, the loss function in the process of training the multimodal visual speech navigation model based on the diversified data set is:

[0029] The loss function during training is:

[0030] L=λL int +(1-λ)(L nav +γL kg )L (5)

[0031] Among them, L int is the interaction loss, which is used to reflect the interaction between the robot and the environment when performing tasks; L nav is the navigation loss, which is used to measure the accuracy or efficiency of the robot’s navigation strategy. kg is a loss term related to the knowledge graph, which is used to evaluate the accuracy or consistency of the robot when using the knowledge graph for navigation. λ and γ are parameters used to adjust the balance loss.

[0032] Furthermore, the visual encoder outputs RGB features F (RGB), semantic segmentation feature F (s) , deep features F (d) and region proposal feature F (0) .

[0033] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for generating and uploading the trusted state alarm information are implemented.

[0034] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for generating and sending trusted state alarm information are realized.

[0035] The present invention has the following beneficial effects:

[0036] During specific operation, the substation visual voice navigation method, system, device and medium described in the present invention deeply fuses different visual information with voice information through multimodal aggregation technology, thereby generating a more comprehensive and accurate description of the navigation environment. Experimental results show that compared with the traditional visual navigation method that relies solely on RGB information, the present invention can still maintain high navigation accuracy and stability in the complex scenes of substations. It should be noted that the core of the present invention lies in its unique multimodal information processing capability. By integrating different visual observation information and voice commands, the robot's depth of understanding of the surrounding environment and the accuracy of navigation tasks are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2 This is a model architecture diagram in the present invention;

[0040] Figure 3 is a schematic diagram of the visual module;

[0041] Figure 4 is a schematic diagram of an auditory encoder;

[0042] Figure 5 A schematic diagram of a multi-processor module. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0045] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0046] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0047] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0048] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0051] Embodiment 1

[0052] refer to Figure 1 The substation visual voice navigation method of the present invention comprises:

[0053] 1) Build a dataset;

[0054] Collect real scene data of 220kV substations, including the building layout inside the substation, equipment distribution, types of various equipment, such as transformers, circuit breakers, disconnectors, etc., and the specific location of each equipment in the substation, including equipment number, location, adjacent equipment, etc. Based on this data, build a scene-level knowledge graph with scenes and objects as nodes and the relationships between them as edges.

[0055] 2) Construct a multimodal visual and speech navigation model;

[0056] It should be noted that completing the visual voice navigation task in the substation faces the problem of complex and changeable environment. Most of the existing navigation methods analyze single visual information, and the trained models are sometimes difficult to adapt to some similar complex environments. The present invention adopts multimodal aggregation technology, designs a multi-processor module, integrates voice target information into the network, and also integrates the spatial and semantic information obtained through vision, which can realize multimodal fusion and attention mechanism, so that it can complete the navigation task well.

[0057] Specifically, refer to Figure 2 , Figure 3 , Figure 4 and Figure 5 The multimodal visual speech navigation model includes an auditory encoder, a visual encoder, a multi-processor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multi-processor module, the multi-processor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer.

[0058] 21) The operation process of the visual encoder is:

[0059] The currently observed visual information is input into the following four different Visual Encoders to generate corresponding visual features from the observations of the robot camera:

[0060] RGB feature F (RGB) , using ResNet-18 pre-trained on ImageNet, extracting general image features from the conv5_4 layer;

[0061] Semantic segmentation feature F (s) , Deeplab is used to extract segmentation features. The feature dimension of the last convolutional layer is (2048, 38, 38). The two-dimensional and three-dimensional features are averaged to obtain a 2048-dimensional vector.

[0062] Deep feature F (d) ,Using MonoDepth, through training on Cityscapes and KITTI, a deep feature map with shape (384,512) is extracted;

[0063] Region Proposal Feature F (0) , Faster-RCNN detector with ResNet-50 backbone is used for region proposal and feature extraction.

[0064] 22) The operation process of the auditory encoder is:

[0065] To convert speech to text, a new open source speech recognition model called DeepSpeech is used. Then to get the target from the command, the spaCy package (Natural Language Toolkit) is used to identify the noun words in the text, and then GloVe is used to get the word vector of the target object word.

[0066] 23) The specific operations of the multi-processor module are:

[0067] Vision and hearing are two different perceptual modalities, and information from various modalities can be analyzed from visual observations. Before aggregation, these different features are converted into tensors, such as Figure 4 As shown in the figure. Since the value distribution and shape of each feature are different, different multi-processor modules are used to process different forms of weighted feature maps. In order to aggregate all tensors, they are concatenated and input into a point convolution layer. The result is then input into a Flatten layer to convert it into a one-dimensional vector, thus obtaining the final embedding.

[0068] The embedding obtained by the multimodal aggregation method can be expressed as:

[0069] e i =γ(F i (RGB) ,F i (s) , F i (d) , F i (o) , B (o) ) (1)

[0070] Need to process a feature map

[0071]

[0072] When t = 0, the feature map is a d 0 dimensional vector, the conversion module uses a linear layer, and the aggregatable tensor converted from the vector is:

[0073] T = ReLU(F·W Linear +B Linear )(3)

[0074] Among them, the rectified linear unit (ReLU) is used as the activation function.

[0075] When t=2 or t=3, the aggregatable tensor obtained in the remaining two dimensions through one or more convolutional layers is:

[0076]

[0077] in, Represents the convolution structure of the layer. When initializing these convolution layers, the weights are multiplied by a scaling factor, which is recommended to be

[0078] 23) The specific operations of the output layer are:

[0079] The aggregation results are sent to the decoder for encoding, and the output of the decoder is aligned with the scene-level knowledge graph output by the knowledge graph module to generate executable instructions for the robot.

[0080] 3) using the diverse dataset to train a multimodal visual speech navigation model;

[0081] The loss function during training is:

[0082] L=λL int +(1-λ)(L nav +γL kg )L (5)

[0083] Among them, L int is the interaction loss, which is used to reflect the interaction between the robot and the environment when performing tasks, and serves as a self-supervisory signal to guide the adjustment of navigation strategies; L nav is the navigation loss, which is used to measure the accuracy or efficiency of the robot’s navigation strategy. kg is a loss term related to the knowledge graph, which is used to evaluate the accuracy or consistency of the robot when using the knowledge graph for navigation. λ and γ are parameters used to adjust the balance loss.

[0084] During training, a diverse data set containing visual images of the substation environment and corresponding voice commands is prepared. A sub-goal-based supervised learning method is adopted. By taking several intermediate positions that the robot passes through when reaching the target position as sub-goals, the robot obtains additional positive rewards when it reaches the sub-goal, thereby accelerating the training of the active navigation strategy model and improving training efficiency.

[0085] 4) Obtain visual information and voice commands of the current environment;

[0086] 5) Inputting the visual information and voice instructions of the current environment into the trained multimodal visual and voice navigation model to obtain robot executable instructions, and controlling the robot according to the robot executable instructions until the target position is reached.

[0087] Confirmatory trials

[0088] To compare the performance, 200 different events were evaluated. Table 1 shows the validation results of the proposed method and the benchmark method.

[0089] From the index results, it can be concluded that the present invention is superior to the baseline method that only uses RGB features. Experiments show that the present invention can improve the navigation success rate (Success).

[0090] Table 1

[0091]

[0092] It should be noted that the present invention adds multimodal visual information to the navigation system to promote the understanding of the environment, so as to better complete the navigation task. The results show that compared with the most advanced baseline method, the present invention is superior in terms of success rate. In contrast, the advantages of the present invention are mainly reflected in the following aspects: the addition of multimodal visual information can enable the robot to adapt well to the complex scenes of the substation: the extraction of rich depth information helps the robot understand the shape, size, position and other spatial relationships of the equipment in the substation, so as to more accurately judge the surrounding environment; by semantically segmenting the images and giving them corresponding semantic labels, the accuracy of navigation is further improved, and it has stronger robustness. The construction of the scene-level knowledge graph effectively guarantees the accuracy of global navigation; the present invention can achieve precise navigation, which can assist substation operation and maintenance personnel in substation faulty equipment, and can bring greater benefits.

[0093] Embodiment 2

[0094] The substation visual voice navigation system of the present invention comprises:

[0095] The acquisition module is used to obtain visual information and voice commands of the current environment;

[0096] The navigation module is used to input the visual information of the current environment and the voice instructions into the trained multimodal visual voice navigation model to obtain the robot executable instructions, and control the robot according to the robot executable instructions until it reaches the target position. The multimodal visual voice navigation model includes an auditory encoder, a visual encoder, a multi-processor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multi-processor module, the multi-processor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer.

[0097] As an embodiment of the present invention, it also includes:

[0098] A first building module is used to build a diverse dataset containing visual images of the substation environment and corresponding voice commands;

[0099] The second building module is used to build a multimodal visual speech navigation model;

[0100] A training module is used to train the multimodal visual-speech navigation model based on the diversified data set to obtain a trained multimodal visual-speech navigation model.

[0101] As an implementation mode of the present invention, the loss function in the process of training the multimodal visual speech navigation model based on the diversified data set is:

[0102] The loss function during training is:

[0103] L=λL int +(1-λ)(L nav +γL kg )L (5)

[0104] Among them, L int is the interaction loss, which is used to reflect the interaction between the robot and the environment when performing tasks; L nav is the navigation loss, which is used to measure the accuracy or efficiency of the robot’s navigation strategy. kg is a loss term related to the knowledge graph, which is used to evaluate the accuracy or consistency of the robot when using the knowledge graph for navigation. λ and γ are parameters used to adjust the balance loss.

[0105] As an embodiment of the present invention, the visual encoder outputs RGB feature F (RGB) , semantic segmentation feature F (s) , deep features F (d) and region proposal feature F (0) .

[0106] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0107] Embodiment 3

[0108] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the substation visual voice navigation method are implemented, for example, including: constructing a diversified data set including visual images of the substation environment and corresponding voice instructions; constructing a multimodal visual voice navigation model; training the multimodal visual voice navigation model based on the diversified data set to obtain a trained multimodal visual voice navigation model; obtaining visual information and voice instructions of the current environment; inputting the visual information and voice instructions of the current environment into the trained multimodal visual voice navigation model to obtain robot executable instructions, and controlling the robot according to the robot executable instructions until the target position is reached. The multimodal visual voice navigation model comprises an auditory encoder, a visual encoder, a multiprocessor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multiprocessor module, the multiprocessor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, network interface, and memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0109] Embodiment 4

[0110] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of implementing the substation visual voice navigation method, for example, include: constructing a diversified data set containing a visual image of the substation environment and corresponding voice instructions; constructing a multimodal visual voice navigation model; training the multimodal visual voice navigation model based on the diversified data set to obtain a trained multimodal visual voice navigation model; obtaining visual information and voice instructions of the current environment; inputting the visual information and voice instructions of the current environment into the trained multimodal visual voice navigation model to obtain a robot executable instruction, and controlling the robot according to the robot executable instruction until reaching the target position, wherein the multimodal visual voice navigation model includes an auditory encoder, a visual encoder, a multiprocessor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multiprocessor module, the multiprocessor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer. Specifically, the computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0115] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0116] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0117] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A substation visual voice navigation method, characterized in that: include: Get visual information of the current environment and voice commands; The visual information of the current environment and the voice commands are input into the trained multimodal visual voice navigation model to obtain executable commands for the robot, and the robot is controlled according to the executable commands until the target position is reached. The multimodal visual voice navigation model includes an auditory encoder, a visual encoder, a multiprocessor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multiprocessor module, the multiprocessor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer.

2. The substation visual voice navigation method according to claim 1 is characterized in that: Before inputting the visual information and voice instructions of the current environment into the trained multimodal visual voice navigation model, the method further includes: Build a diverse dataset containing visual images of substation environments and corresponding voice commands; Construct a multimodal visual and speech navigation model; The multimodal visual-speech navigation model is trained based on the diversified data set to obtain a trained multimodal visual-speech navigation model.

3. The substation visual voice navigation method according to claim 2 is characterized in that: The loss function in the process of training the multimodal visual speech navigation model based on the diversified data set is: The loss function during training is: L=λL int +(1-λ)(L nav +γL kg )L (5) Among them, L int is the interaction loss, which is used to reflect the interaction between the robot and the environment when performing tasks; L nav is the navigation loss, which is used to measure the accuracy or efficiency of the robot’s navigation strategy. kg is a loss term related to the knowledge graph, which is used to evaluate the accuracy or consistency of the robot when using the knowledge graph for navigation. λ and γ are parameters used to adjust the balance loss.

4. The substation visual voice navigation method according to claim 1, characterized in that: The visual encoder outputs RGB features F (RGB) , semantic segmentation feature F (s) , deep features F (d) and region proposal feature F (0) .

5. A substation visual voice navigation system, characterized in that: include: The acquisition module is used to obtain visual information and voice commands of the current environment; The navigation module is used to input the visual information of the current environment and the voice instructions into the trained multimodal visual voice navigation model to obtain the robot executable instructions, and control the robot according to the robot executable instructions until it reaches the target position. The multimodal visual voice navigation model includes an auditory encoder, a visual encoder, a multi-processor module, a decoder, a knowledge graph module and an output layer, wherein the auditory encoder and the visual encoder are connected to the multi-processor module, the multi-processor module is connected to the decoder, and the decoder and the knowledge graph module are connected to the output layer.

6. The substation visual voice navigation system according to claim 5, characterized in that: Also includes: A first building module is used to build a diverse dataset containing visual images of the substation environment and corresponding voice commands; The second building module is used to build a multimodal visual speech navigation model; A training module is used to train the multimodal visual-speech navigation model based on the diversified data set to obtain a trained multimodal visual-speech navigation model.

7. The substation visual voice navigation system according to claim 6, characterized in that: The loss function in the process of training the multimodal visual speech navigation model based on the diversified data set is: The loss function during training is: L=λL int +(1-λ)(L nav +γL kg )L (5) Among them, L int is the interaction loss, which is used to reflect the interaction between the robot and the environment when performing tasks; L nav is the navigation loss, which is used to measure the accuracy or efficiency of the robot’s navigation strategy. kg is a loss term related to the knowledge graph, which is used to evaluate the accuracy or consistency of the robot when using the knowledge graph for navigation. λ and γ are parameters used to adjust the balance loss.

8. The substation visual voice navigation system according to claim 5, characterized in that: The visual encoder outputs RGB features F (RGB) , semantic segmentation feature F (s) , deep features F (d) and region proposal feature F (0) .

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating and sending trusted status alarm information as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating and uploading trusted status alarm information as described in any one of claims 1 to 4 are implemented.

Images