Energy storage power inspection robot interaction method based on multi-modal semantic understanding

Abstract

The application relates to the technical field of semantic understanding, and particularly discloses a kind of energy storage electric power inspection robot interaction methods based on multi-modal semantic understanding.Methods include: parsing the stress voice instruction of user and extracting the section crossing physical red line to establish safety-intention conflict identification;According to conflict identification triggers chassis speed damping, establishes the observation option set containing multiple perspectives;Output to the user compensatory observation option set and capture the ideal observation point actually selected, extract spatial offset parameters and carry out intention feedback analysis, obtain intention feedback characteristics;Along the physical safety boundary outside contour generates micro-cruise trajectory, executes multi-perspective alternate observation and establishes observation confirmation point array;Combine safety boundary and node array to identify key observation trigger area, establish dynamic regulation strategy of autonomous inspection.The application is beneficial to convert the psychological changes of human in response to emergency state into the regulation strategy of chassis navigation map, realizes dynamic intelligent adjustment of robot.

Description

Technical Field

[0001] This invention relates to the field of semantic understanding technology, and specifically to an interaction method for an energy storage power inspection robot based on multimodal semantic understanding. Background Technology

[0002] With the development of smart grids, power inspection robots are widely used in high-risk environments such as substations. In actual operation, when equipment malfunctions, on-site maintenance personnel are often in a state of high stress, and their voice intervention commands often carry the intention of urgently approaching the target, ignoring hidden physical safety boundaries such as energized distances and blind spots.

[0003] Existing robot interaction control systems typically employ rigid interception strategies at their underlying safety mechanisms in such emergency situations. When a manually issued movement command is detected as crossing a physical boundary, the robot immediately halts execution and reports an error. This direct rejection, lacking a compensatory mechanism, not only interrupts the hazard identification process but also requires personnel to manually re-find a safe perspective and issue commands again, reducing emergency response efficiency. Furthermore, existing systems cannot perceive and quantify the user's stress state and psychological intention deviations when yielding to danger, preventing the robot from translating human attention and tension in the face of sudden danger into data features for map navigation. Consequently, in subsequent autonomous inspections, the robot cannot dynamically adjust the observation density and patrol trajectory of high-risk areas based on the degree of human intervention in previous high-risk events, hindering the intelligent adaptive learning of inspection strategies.

[0004] Therefore, this invention provides an interaction method for energy storage power inspection robots based on multimodal semantic understanding. Summary of the Invention

[0005] The purpose of this invention is to provide an interaction method for energy storage power inspection robots based on multimodal semantic understanding, so as to solve the above-mentioned background problems.

[0006] The objective of this invention can be achieved through the following technical solutions: An interaction method for energy storage power inspection robots based on multimodal semantic understanding includes the following steps: The power inspection robot analyzes the user's stressed voice commands and establishes the user's expected target movement path; it then performs a collision comparison between the target movement path and the physical safety boundary of the environment to obtain the sections where the path crosses the physical red line and establishes corresponding safety-intent conflict indicators. Based on the safety-intent conflict identifier, the speed reduction damping of the power inspection robot chassis is controlled, while scanning the safety environment around the conflict zone, screening out candidate parking points of observable targets, and establishing a set of observation options including multiple perspectives. Output a set of compensatory observation options to the user and capture the ideal observation point actually selected by the user; extract the spatial offset parameters between the ideal observation point and the initially expected target movement path; perform intent feedback parsing based on the spatial offset parameters and organize the intent feedback features. Extract the residual tension value of intent from the intent deviation feedback features, analyze the number of observation compensation nodes, and generate a micro-cruise trajectory around the target along the outer contour of the physical safety boundary; perform multi-view alternating observations on the micro-cruise trajectory to establish an array of observation confirmation points.

[0007] Furthermore, the security-intent conflict identifier is established as follows: By retrieving the equipment geographic information system ledger and static topology map, a spatial index grid for the robot is established; Obtain the spatial trajectory coordinate sequence of the target's movement path, compare and calculate the spatial position of each coordinate point in the spatial trajectory coordinate sequence with the three-dimensional safety protection surface, and determine whether the target's movement path crosses the physical safety boundary and intrudes into the internal space on the side of the hazard source. If the judgment result is that there is an intrusion, then it is determined that all continuous coordinate points located inside the physical security boundary are extracted sequentially from the spatial trajectory coordinate sequence, and the segments that are fitted to overlap in space are extracted as the trajectory overlap segments. Extract the start coordinates, end coordinates, and intrusion depth parameters of the overlapping segments, generate an identifier field, and define it as a security-intent conflict identifier.

[0008] Furthermore, the method for screening the candidate stopping points is as follows: By scanning the field of view around the current location using a power inspection robot, the surrounding terrain data and the coordinates of static obstacles outside the physical safety boundary can be obtained. Based on the surrounding terrain data and the coordinates of static obstacle distribution, multiple safe-passing surrounding physical grids are delineated as alternative areas in the spatial index grid structure. For each candidate area, obtain the spatial coordinates within the candidate area and analyze the line-of-sight connectivity from the spatial coordinates to the abnormal area of ​​the target device; Spatial coordinate points with expected line-of-sight connectivity are extracted through spatial clustering to form candidate stopping points for effective visual coverage.

[0009] Furthermore, the method for analyzing the line-of-sight connectivity is as follows: Define the target equipment observation surface for the abnormal area of ​​the target equipment, and construct a spatial straight line connecting the spatial coordinate points of the candidate area and the target equipment observation surface in the three-dimensional coordinate system; If the spatial straight line does not intersect or cut off any entity structure coordinates in the static topology diagram, then the line-of-sight connectivity is determined to be as expected.

[0010] Furthermore, the method for generating the compensatory observation option set is as follows: Extract the predicted observation angle, the relative observation distance to the target device, and the spatial angle between the predicted observation angle and the frontal normal vector of the target device corresponding to all candidate parking points. Establish a comprehensive evaluation rule, sort all candidate parking points, and obtain a sorted list. The candidate stopping points in the sorted list are truncated, and their spatial location parameters and viewpoint features are extracted and combined to form a compensatory observation option set.

[0011] Furthermore, the method for performing the intent feedback parsing is as follows: Extract the spatial concession coefficient for the degree of physical retreat; The peak sound pressure level of the secondary voice signal when the user issues the final command is called, and the ratio of the difference between the peak sound pressure level and the peak sound pressure level of the original stressed voice command is calculated to obtain the stress attenuation rate. Intent residual tension is assessed based on the original stress speech sound pressure peak value and spatial concession coefficient to obtain residual tension value; The intended residual tension value is integrated with the actual observation point coordinates to serve as the intended deviation feedback feature.

[0012] Furthermore, the method for conducting the intentional residual tension assessment is as follows: The peak sound pressure level of the original stressed speech was extracted as the initial intention potential energy, and the spatial concession coefficient and stress attenuation rate were respectively used as two characteristic parameters affecting tension release. By combining two characteristic influencing parameters to perform attenuation analysis on the initial intention potential energy, the residual tension value characterizing the user's internal resistance intensity is obtained.

[0013] Furthermore, the micro-cruising trajectory is generated as follows: Based on the intention deviation feedback characteristics, a node tension multiplication analysis was performed to obtain the total number of observed compensation nodes; With the center coordinates of the abnormal area of ​​the target equipment as the center and the safety clearance distance corresponding to the physical safety boundary as the radius, construct a safety arc facing the abnormal area of ​​the target equipment. Extract the projected coordinates of the actual selected observation point on the safety arc, and use the projected coordinates as the starting anchor point of the micro-cruise trajectory; Based on the starting anchor point, and according to the pre-set fixed included angle step size, the system alternately and symmetrically expands to both sides along the safety arc, and sequentially intercepts a number of spatial coordinate points equal to the total number of observation compensation nodes, forming a distributed observation node array. All spatial coordinate points in the distributed observation node array are smoothly connected sequentially according to their spatial proximity to generate a micro-cruise trajectory around the abnormal area of ​​the target device.

[0014] Furthermore, the residual tension value of intent in the intent deviation feedback features and the distributed observation node array are extracted and combined with the physical safety boundary to identify key observation trigger areas; at the same time, a dynamic control strategy for autonomous inspection within the key observation trigger areas is established.

[0015] Furthermore, the method for identifying the key observation trigger area is as follows: Extract the residual tension value of intent stored in the intent deviation feedback feature, and retrieve the three-dimensional coordinate set of all final observation confirmation points contained in the distributed observation node array; In the spatial index grid, based on the physical contour formed by the set of three-dimensional coordinates and the micro-cruise trajectory, an outward spatial three-dimensional grid expansion operation is performed. The intentional residual tension value is rounded down and converted into the number of mesh expansion layers for the spatial three-dimensional mesh expansion operation; After completing the spatial three-dimensional mesh expansion operation based on the number of mesh expansion layers as the expansion distance, a three-dimensional spatial mesh block is generated outside the physical safety boundary of the abnormal area of ​​the target device, which is defined as the key observation trigger area.

[0016] The beneficial effects of this invention are as follows: By establishing a collision comparison mechanism between the target movement path and the physical safety boundary of the environment, the expected trajectory that crosses the physical red line is intercepted at the bottom control end. Unstructured stress voice commands are converted into safety-intent conflict identifiers in the spatial coordinate system. Physical protection blocking response is executed before the system outputs motion commands, thereby improving the spatial collision tolerance of the equipment in complex obstacle distribution environments.

[0017] A spatial scanning calculation strategy integrating chassis deceleration damping and line-of-sight connectivity analysis is constructed. While restricting out-of-bounds motion parameters, it triggers a coordinate search of the safe navigation grid, extracting candidate parking points without physical obstructions and satisfying the effective field of view width. This generates a multi-view compensatory observation set, continuing the execution logic of the abnormal equipment investigation task while maintaining underlying physical safety constraints. By extracting the spatial yaw angle and retreat distance between the actual selected observation point and the initial expected target point, and combining this with the peak sound pressure change ratio of the secondary command for tension analysis, the human-machine control game process is transformed into structured residual tension numerical parameters, enabling the quantitative processing of the user's stress intervention state.

[0018] By executing an observation node multiplication mapping algorithm based on intention residual tension values, a spatial coordinate array is dynamically calculated along an arc surface limited by a safe retreat distance, forming a distributed micro-cruise trajectory. A servo mechanism then performs multi-pose fixed-point image acquisition actions along this trajectory, converting high-stress state parameters into multi-viewpoint focusing scheduling commands at the underlying hardware level, thus increasing the 3D image acquisition data source for local anomaly areas. An intention deviation tension value-driven 3D spatial mesh expansion algorithm is used, with tension values ​​as distance parameters to generate a key observation trigger area mesh outside the original physical contour, which is then written into the global navigation topology. When the device subsequently performs routine inspections and crosses the corresponding mesh boundary, it autonomously triggers task truncation, which is beneficial for iterating manual intervention data under single abnormal operating conditions into long-term navigation maps and observation density control strategies. Attached Figure Description

[0019] The invention will now be further described with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart of an interaction method for an energy storage power inspection robot based on multimodal semantic understanding, according to the present invention. Figure 2 This is a flowchart of the intentional residual tension assessment in this invention; Figure 3 This is a functional block diagram of an interactive system for a power inspection robot based on multimodal semantic understanding, as described in this invention. Detailed Implementation

[0021] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0022] Example 1:

[0023] like Figure 1 As shown, an interaction method for an energy storage power inspection robot based on multimodal semantic understanding includes the following steps: S10: The power inspection robot analyzes the user's stressed voice commands and establishes the user's expected target movement path; it performs collision comparison processing between the target movement path and the physical safety boundary of the environment to obtain the section where the path crosses the physical red line and establishes the corresponding safety-intent conflict identifier. The power inspection robot interprets the user's stressed voice commands and establishes the user's expected target movement path in the following way: In some embodiments, the parsing unit of the power inspection robot is invoked to receive the original voice input from the user (e.g., maintenance personnel) at the power operation site, and to extract the sound pressure peak, speech rate variation amplitude and frequency energy distribution information of the original voice input. Based on the extracted peak sound pressure level and the amplitude of speech rate variation, speech commands are analyzed to identify stressful speech commands that convey urgency and an intent to control an emergency. It should be noted that the method for parsing voice commands is as follows: extract the global maximum value of the sound pressure peak in the original voice input, detect the time interval between adjacent effective speech segments to obtain the speech rate change rate, preset the stable sound pressure benchmark value of the environment (such as 65 dB), and when the sound pressure peak exceeds the high stress sound pressure threshold (such as 90 dB) and the speech rate change rate is greater than 20%, it is determined that the user is in a high stress state, and the current voice input is marked as a stress voice command; The method for detecting the time interval between adjacent valid speech segments is as follows: using the existing endpoint detection algorithm, the original speech waveform is divided into silent interval segments and valid speech segments, and the time span between the boundaries of two consecutive valid speech segments is taken as the time interval. The system performs preprocessing on the response to the voice command and outputs the voice enhancement signal. The preprocessing includes compressing the resonance region of the original speech input signal, bandpass enhancement of the transition band, and suppression of background interference bands. The speech enhancement signal, together with the contextual task description information, is input into the existing speech recognition component to extract the direction, distance, and target device number contained in the interaction command information as spatial semantic information; It should be noted that the speech recognition component uses publicly available speech recognition algorithms in the relevant field to extract acoustic features and directly converts the acoustic features into text sequences. Using the three-dimensional coordinate position and spatial semantic information currently collected by the pose sensor of the power inspection robot, the motion planning module of the power inspection robot is invoked, and the target movement path expected by the user is generated with the user's command intention as the target. The process of performing collision comparison between the target's movement path and the physical safety boundary of the environment to obtain the sections where the path crosses the physical red line and establishing corresponding safety-intent conflict identifiers is as follows: By retrieving the equipment GIS (Geographic Information System) ledger and static topology map of the power grid area, a three-dimensional spatial index grid of the robot's current power grid environment is established; Establish physical safety boundaries in the spatial index grid, perform offline simulation projection of the generated target movement path in the spatial index grid, and obtain the spatial trajectory coordinate sequence of the target movement path; It should be noted that the physical safety boundary is established as follows: based on the operating level and voltage level of different power grid equipment, the discharge danger distance of energized high-voltage equipment, the physical shielding blind zone of high-voltage tower base, and the identified structural anomaly areas (such as the fracture of the load-bearing base) are taken as the physical red line that cannot be crossed. Based on the physical red line, a three-dimensional safety protection surface is constructed in the spatial coordinate system to isolate the hazard source, which serves as the physical safety boundary of the power inspection robot. The spatial position of each coordinate point in the spatial trajectory coordinate sequence is compared and calculated with the three-dimensional safety protection surface to determine whether the target's movement path crosses the physical safety boundary and intrudes into the internal space on the side of the hazard source. If the judgment result is that there is an intrusion, it is determined that the expected path crosses the physical red line. All continuous coordinate points located inside the physical security boundary are extracted sequentially from the spatial trajectory coordinate sequence, and the segments that are fitted to overlap in space are extracted as the trajectory overlap segments. If the judgment result is that the physical security boundary has not been violated, the changes of each coordinate point in the spatial trajectory coordinate sequence will be continuously monitored. Extract the starting point coordinates, ending point coordinates, and intrusion depth parameters of the overlapping segments, and combine them with the physical security boundary attributes corresponding to the overlapping trajectory segments to generate an identifier field representing high-risk actions, which is defined as a security-intent conflict identifier.

[0024] S20: Based on the safety-intent conflict identifier, control the deceleration damping of the power inspection robot chassis, scan the safety environment around the conflict section, screen out candidate parking points of observable targets, and establish a set of observation options including multiple perspectives. The method for controlling the deceleration damping of the power inspection robot chassis based on the safety-intent conflict identifier is as follows: After establishing a safety-intent conflict identifier, the control unit of the power inspection robot is invoked to intercept the underlying motion control commands that originally corresponded to the expected target movement path; Obtain the current speed of the power inspection robot, and the remaining straight-line distance from the robot's current physical center to the starting point of the trajectory overlap section; By combining the driving speed and the remaining straight distance, the braking torque required by the chassis drive components is calculated, and the corresponding control signal is output to limit the chassis speed until the power inspection robot is in a hovering and ready state. The method for scanning the security environment surrounding the conflict zone, selecting candidate stopping points where observable targets can be found, and establishing a set of observation options with multiple perspectives is as follows: When the power inspection robot is in a hovering and ready state, the robot's data acquisition unit is invoked. Through the onboard image acquisition and ranging components, the robot performs a field-of-view spatial scan around the current position to obtain the surrounding terrain data and static obstacle distribution coordinates outside the physical safety boundary. Based on the surrounding terrain data and the coordinates of static obstacle distribution, multiple safe-passing surrounding physical grids are delineated as alternative areas in the spatial index grid structure. The method for defining multiple safe-passing surrounding physical grids is as follows: outside the physical safety boundary, extract the slope characteristic parameters and surface flatness parameters of the corresponding ground in the spatial index grid structure, and mark the grids with slopes lower than the maximum climbing angle limit of the power inspection robot and no static obstacles with heights exceeding the ground clearance of the chassis as safe-passing surrounding physical grids. For each candidate area, obtain the spatial coordinates of the candidate area and analyze the line-of-sight connectivity from the spatial coordinates to the abnormal area of ​​the target device, and eliminate the blind zone coordinates that have physical obstruction or a field of view angle less than the set observation threshold. Among them, the abnormal area of ​​the target equipment is a three-dimensional spatial area with structural damage, abnormal heating and other characteristics identified by comparing the equipment GIS ledger with the previous routine inspection images; For example, the observation threshold is set as follows: obtain the minimum effective resolution parameter of the image acquisition component carried by the power inspection robot, and combine it with the physical size of the abnormal area of ​​the target device to be observed to deduce the limit observation angle that meets the requirements for clear image recognition, and set the limit observation angle as the observation threshold; if the field of view is less than the observation threshold, it is determined that the acquired image has too much distortion or insufficient effective pixels, and cannot meet the compensation observation requirements. The preferred method for analyzing line-of-sight connectivity is: Define the target equipment observation surface for the abnormal area of ​​the target equipment, and construct a spatial straight line connecting the spatial coordinate points of the candidate area and the target equipment observation surface in the three-dimensional coordinate system; Among them, the target device's observation surface is the outer surface of the three-dimensional space region that is not obstructed by static obstacles and faces the robot's currently walkable grid. Mathematically, it is represented by the surface's normal vector and the center coordinate point. If the spatial straight line does not intersect or cut off any entity structure coordinates in the static topology diagram, then the line-of-sight connectivity is determined to be as expected. If a straight line in space intersects with the coordinates of an entity in the static topology diagram, the target device is determined to be in a blind spot where the line of sight does not meet the requirements, and the spatial coordinate point is directly removed from the candidate area. Spatial coordinate points with expected line-of-sight connectivity are spatially clustered and extracted to form multiple candidate stopping points that can achieve effective visual coverage without crossing physical red lines. The method for spatial clustering and extracting spatial coordinate points with good line-of-sight connectivity is as follows: based on the physical connectivity distance between adjacent spatial coordinate points, a clustering radius is set to represent the physical area occupied by the robot for safe parking; taking any spatial coordinate point with good line-of-sight connectivity as the center, all adjacent spatial coordinate points falling within the clustering radius are divided into the same coordinate cluster; the spatial geometric center coordinates of each coordinate cluster are extracted as an independent candidate parking point representing the local area. The method for generating a compensatory observation option set that includes multiple perspectives is as follows: By connecting the spatial coordinates of the candidate parking point with the geometric center of the target device's observation surface, a spatial line-of-sight vector is generated. The heading angle and pitch angle of the spatial line-of-sight vector in the three-dimensional coordinate system are extracted and combined to define the predicted observation angle. Extract the predicted observation viewpoint, the relative observation distance to the target device, and the spatial angle between the predicted observation viewpoint and the frontal normal vector of the target device for all candidate parking points; Based on the comprehensive evaluation rules of relative observation distance from near to far and the spatial angle between the predicted observation angle and the frontal normal vector of the target equipment from small to large, all candidate stopping points are sorted to obtain a sorted list; For example, the sorting method based on the comprehensive evaluation rules is as follows: extract the maximum relative observation distance among all current candidate stopping points, and the maximum spatial angle formed with the frontal normal vector of the target device among all predicted observation views; For each candidate stopping point, the relative observation distance is divided by the maximum relative observation distance to obtain the distance normalization value, and the spatial angle of the candidate stopping point is divided by the maximum spatial angle to obtain the viewpoint normalization value. Based on the observation focus requirements of the abnormal area of ​​the target equipment, distance weighting coefficients and viewing angle weighting coefficients are set for the distance normalized value and the viewing angle normalized value, respectively. For example, the distance weighting coefficient and the viewing angle weighting coefficient are set as follows: if the anomaly type of the target device is a minor structural crack, and the determination system requires image clarity more than image orthogonality, then the distance weighting coefficient is set to 0.7 and the viewing angle weighting coefficient is set to 0.3; if the anomaly type is overall device deformation, and the determination system requires image orthogonality viewing angle more than distance, then the distance weighting coefficient is set to 0.4 and the viewing angle weighting coefficient is set to 0.6. The distance cost is obtained by multiplying the normalized distance value by the distance weight coefficient, and the viewpoint cost is obtained by multiplying the normalized viewpoint value by the viewpoint weight coefficient. The distance cost and the viewpoint cost are added together to obtain the observation cost score for each candidate stopping point; All candidate stopping points are sorted in ascending order based on their comprehensive observation cost scores, from smallest to largest. Extract several candidate stopping points located at the target location from the sorted list, and combine and encapsulate the spatial location parameters and viewpoint features of the candidate stopping points into a set of compensatory observation options; Preferably, the method for selecting several candidate stopping points located at the target position is as follows: set an upper limit value for the number of observation options, for example, set the upper limit value to 3; extract the three candidate stopping points with the lowest observation cost scores in ascending order; if the total number of candidate stopping points is less than three, extract all candidate stopping points as a set of compensatory observation options. Among them, the spatial position parameter specifically refers to the three-dimensional coordinates of the candidate parking point in the spatial index grid; the viewpoint feature specifically refers to the pose parameters required by the gimbal equipped with the image acquisition component when it is aligned with the observation surface of the target device, namely the heading angle and pitch angle. The control unit is invoked to convert the compensatory observation option set into structured voice feedback content, which is then broadcast to maintenance personnel through a speaker to guide users to reissue safe observation location scheduling commands.

[0025] Example 2:

[0026] Please see Figure 1 As shown, an interaction method for an energy storage power inspection robot based on multimodal semantic understanding includes the following steps: S30: Output a set of compensatory observation options to the user and capture the ideal observation point actually selected by the user; extract the spatial offset parameters between the ideal observation point and the initially expected target movement path; perform intent feedback parsing based on the spatial offset parameters and organize the intent feedback features. The method for outputting the set of compensatory observation options to the user and capturing the ideal observation point actually selected by the user is as follows: The voice interaction module of the power inspection robot outputs the serial number and relative observation angle information of each candidate stopping point in the compensation observation option set to the user in the form of voice broadcast. At the same time, the robot's audio acquisition array is turned on, and a voice listening window of a preset duration (e.g., 5 seconds) is set to collect the voice signals fed back by the user in the listening window; Use existing speech recognition components to perform text parsing on speech signals and extract ordinal numbers or locative words from the text. The extracted ordinal or directional information is semantically compared with each candidate stopping point in the compensatory observation option set to determine the uniquely matching candidate stopping point as the ideal observation point actually selected by the user. The method for extracting the spatial offset parameter between the ideal observation point and the initially expected target movement path is as follows: In a three-dimensional coordinate system, obtain the physical center coordinates of the power inspection robot when it triggers the safety-intent conflict flag, and use them as the origin coordinates. Retrieve the coordinates of the terminal target in the initially expected target movement path, and construct the initial target space vector with the origin coordinates as the starting point and the terminal target coordinates as the ending point. Extract the three-dimensional spatial coordinates of the observation point actually selected by the user, and construct the actual execution space vector with the origin coordinates as the starting point and the three-dimensional spatial coordinates of the observation point as the ending point; Calculate the magnitudes of the initial intention space vector and the actual execution space vector respectively. Divide the dot product of the two vectors by the product of their magnitudes to obtain the cosine quotient. Calculate the inverse cosine function of the cosine quotient and obtain the angle in three-dimensional space, which is used as the spatial yaw angle. At the same time, the Euclidean distance calculation formula is used to calculate the absolute straight-line distance between the three-dimensional spatial coordinates of the observation point actually selected by the user and the terminal target coordinates, which is used as the safety clearance distance; The spatial yaw angle and the safe clearance distance are combined to form a set of parameters that include angle offset attributes and distance offset attributes, which serve as the spatial offset parameters; The method for parsing intent feedback based on spatial offset parameters and obtaining intent feedback features is as follows: Extract the spatial yaw angle and safe retreat distance from the spatial geometric offset parameters, divide them by the maximum parameter value in the corresponding candidate set, normalize them, and then sum them by weight to calculate the spatial concession coefficient that characterizes the degree of physical retreat. It should be noted that, based on the principle of prioritizing on-site safety, the weighting coefficient for the safety retreat distance (e.g., 0.6) is set to be greater than the weighting coefficient for the spatial yaw angle (e.g., 0.4), and then a weighted sum is performed. The audio acquisition module is called to extract the peak sound pressure level of the secondary speech signal when the user issues the final command. The ratio of the difference between the peak sound pressure level and the peak sound pressure level of the original stress speech command is calculated to obtain the stress attenuation rate, which represents the emotional recovery status. The residual tension of intent is evaluated based on the peak sound pressure level of the original stressed speech and the spatial concession coefficient, so as to obtain the residual tension value of the user's internal resistance intent. Preferred, such as Figure 2 As shown, the method for assessing the intended residual tension is as follows: S301. Extract the peak sound pressure level of the original stressed speech as the initial intention potential energy, and use the spatial concession coefficient and stress attenuation rate as two characteristic parameters affecting tension release, respectively. S302. Perform attenuation analysis on the initial intention potential energy by combining two characteristic influence parameters to obtain the residual tension value characterizing the user's internal resistance intensity; Preferably, the method for performing attenuation analysis is as follows: the spatial concession coefficient is used as the deformation ratio of the intention release, the stress attenuation rate is used as the emotional dissipation coefficient, a preset dissipation conversion base with the same dimension as the initial intention potential energy is introduced, the emotional dissipation coefficient is multiplied by the dissipation conversion base to calculate the energy dissipation; the initial intention potential energy is multiplied by the attenuation multiplier and the energy dissipation is subtracted to calculate the residual tension value. Meanwhile, to avoid computational overflow, the deformation ratio is added to a preset positive smoothing constant and the reciprocal is taken as the attenuation multiplier. The preset positive smoothing constant is set to a value between 0.0001 and 0.05. In a specific embodiment of the present invention, to balance the sensitivity and calculation accuracy of the system's collision avoidance response, the preset positive smoothing constant is set to 0.01. The residual tension value is calculated by multiplying the initial potential energy by the decay multiplier and subtracting the energy dissipation. The residual tension value of the intention is integrated and encapsulated with the actual observation point coordinates to generate structured data that reflects the state of human-machine safety game, which serves as the final intention deviation feedback feature.

[0027] S40: Extract the residual tension value of intent from the intent deviation feedback features, analyze the number of observation compensation nodes, and generate a micro-cruise trajectory around the target along the outer contour of the physical safety boundary; perform multi-view alternating observations on the micro-cruise trajectory and establish an array of observation confirmation points; The method for extracting the residual tension value of intent from the intent deviation feedback features, analyzing the number of observation compensation nodes, and generating a micro-cruise trajectory around the target along the outer contour of the physical safety boundary is as follows: Extract the continuous intention residual tension value contained in the intention deviation feedback features; Obtain the basic observation base, which is used to reflect the minimum number of observation points required to meet the minimum cross-field requirements of the target equipment's abnormal area under the condition of no emotional spillover. For example, the basic observation base is set as follows: obtain the maximum physical contour width of the abnormal area of ​​the target device and the effective field of view width of the image acquisition component at the current safe backoff distance; divide the maximum physical contour width by the effective field of view width, round the calculated quotient up, and add a preset number of cross-validation node constants (such as 1) to obtain the basic observation base. The residual tension value of the intention is multiplied by the preset discrete mapping ratio and then rounded to obtain the number of tension multiplication nodes used to characterize the amount of emotional overflow. Add the base observation number to the tension multiplication node number and output the total number of observation compensation nodes in integer form; For example, the discrete mapping ratio is set as follows: if the maximum physical safety arc length is 10 meters, the minimum spatial stopping step distance is 1 meter, the theoretical maximum number of observation nodes is 10, and the known basic observation base is 2, then the discrete mapping ratio is set to 8. Using the center coordinates of the abnormal area of ​​the target equipment as the center and the safety clearance distance corresponding to the physical safety boundary as the radius, construct a safety arc facing the abnormal area of ​​the target equipment in three-dimensional space; Extract the projected coordinates of the actual selected observation point on the safety arc, and use the projected coordinates as the starting anchor point of the micro-cruise trajectory; Based on the starting anchor point, and according to a pre-set fixed included angle step (e.g., five degrees), the system alternately and symmetrically expands to both sides along a safety arc, and sequentially intercepts a number of spatial coordinate points equal to the total number of observation compensation nodes, forming a distributed observation node array. All spatial coordinate points in the distributed observation node array are smoothly connected sequentially according to their spatial proximity to generate a micro-cruising trajectory around the abnormal area of ​​the target device. The method for establishing an array of observation confirmation points by performing multi-view alternating observations along the micro-cruise trajectory is as follows: Control the chassis of the power inspection robot to continuously travel along a micro-cruising trajectory; When the power inspection robot reaches each spatial coordinate point in the distributed observation node array, it triggers the chassis to temporarily stop and controls the image acquisition component to perform fixed-point focusing and shooting of the abnormal area of ​​the target equipment; The spatial coordinates corresponding to each fixed-point focusing shot are established as the final observation confirmation point; Multiple local images taken at all final observation confirmation points are compiled and stitched together to form a 3D panoramic image set of the abnormal area of ​​the target device, and the 3D panoramic image set is sent directly to the user's interactive terminal display screen.

[0028] Example 3:

[0029] Please see Figure 1 As shown, an interaction method for an energy storage power inspection robot based on multimodal semantic understanding further includes the following steps: S50: Extract the residual tension value of intent from the intent deviation feedback features and the distributed observation node array, and combine it with the physical safety boundary to identify key observation trigger areas; at the same time, establish a dynamic control strategy for autonomous inspection within the key observation trigger areas; The method for identifying key observation trigger areas by extracting the residual tension value of intent bias feedback features and the distributed observation node array, combined with physical security boundaries, is as follows: Extract the residual tension value of intent stored in the intent deviation feedback feature, and retrieve the three-dimensional coordinate set of all final observation confirmation points contained in the distributed observation node array; In the spatial index grid, based on the physical contour formed by the set of three-dimensional coordinates and the micro-cruise trajectory, an outward spatial three-dimensional grid expansion operation is performed. Obtain the global navigation map of the current environment of the power inspection robot, extract the maximum safe physical distance between the outer side of the physical safety boundary and the nearest static obstacle, divide the maximum safe physical distance by the unit width of the three-dimensional spatial grid, and round down to obtain the maximum number of grid expansion layers; Simultaneously, the theoretical maximum tension boundary value set in the intentional residual tension assessment is obtained; The tension-space mapping coefficient is calculated by dividing the maximum number of mesh expansion layers by the theoretical maximum tension boundary value. After multiplying the intended residual tension value with the tension-space mapping coefficient and normalizing the scale, the product result is rounded up to convert it into the number of mesh expansion layers for the spatial three-dimensional mesh expansion operation. It should be noted that the larger the intentional residual tension value, the more intense the human-machine confrontation, and the farther the physical distance the mesh expands outward. After completing the spatial three-dimensional mesh expansion operation based on the number of mesh expansion layers as the expansion distance, a three-dimensional spatial mesh block with enlarged volume is generated outside the physical safety boundary of the abnormal area of ​​the target device, which is defined as the key observation trigger area. The method for establishing a dynamic control strategy for autonomous inspection within key observation trigger areas is as follows: The geographic coordinates of the key observation trigger area and the target equipment are bound and stored in the global navigation map file of the power inspection robot; The physical center coordinates of the power inspection robot are continuously acquired while it performs subsequent routine automatic inspection tasks. If the physical center coordinates cross and enter the grid boundary of the key observation trigger area, the control components of the power inspection robot will automatically truncate the currently executed point-to-point straight-line navigation task. The power inspection robot forcibly retrieves the bound three-dimensional coordinate set and automatically controls the chassis to move sequentially to all coordinate points in the three-dimensional coordinate set to perform multi-view alternating observation without the need for external voice commands. Understandably, by extending the mesh in physical space by the residual tension of intention, the psychological changes in human response to emergency situations are transformed into the control boundary of the chassis physical navigation map, enabling the robot to dynamically adjust the observation density of high-risk and hidden danger areas.

[0030] Example 4:

[0031] Please see Figure 3 As shown, a power inspection robot interaction system based on multimodal semantic understanding includes the following modules: Identification Establishment Module: Used to analyze the user's stressed voice commands through the power inspection robot and establish the user's expected target movement path; perform collision comparison processing between the target movement path and the physical safety boundary of the environment to obtain the section where the path crosses the physical red line and establish the corresponding safety-intent conflict identifier; Station selection module: It is used to control the deceleration damping of the power inspection robot chassis based on the safety-intent conflict identifier, while scanning the safety environment outside the conflict section, selecting candidate stationing points where observable targets can be observed, and establishing a set of observation options with multiple perspectives. Intent Feedback Module: Outputs a set of compensatory observation options to the user and captures the ideal observation point actually selected by the user; extracts the spatial offset parameters between the ideal observation point and the initially expected target movement path; performs intent feedback parsing based on the spatial offset parameters and organizes the intent feedback features. The observation and analysis module is used to extract the residual tension value of intent from the intent deviation feedback features, analyze the number of observation compensation nodes, and generate a micro-cruise trajectory around the target along the outer contour of the physical safety boundary; it performs multi-view alternating observations on the micro-cruise trajectory and establishes an array of observation confirmation points. Strategy control module: used to extract the residual tension value of intent from the intent deviation feedback features and the distributed observation node array, and combine it with the physical safety boundary to identify key observation trigger areas; at the same time, it establishes a dynamic control strategy for autonomous inspection within the key observation trigger areas.

[0032] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. An interaction method for energy storage power inspection robots based on multimodal semantic understanding, characterized in that, Includes the following steps: The power inspection robot analyzes the user's stressed voice commands and establishes the user's expected target movement path; it then performs a collision comparison between the target movement path and the physical safety boundary of the environment to obtain the sections where the path crosses the physical red line and establishes corresponding safety-intent conflict indicators. Based on the safety-intent conflict identifier, the speed reduction damping of the power inspection robot chassis is controlled, while scanning the safety environment around the conflict zone, screening out candidate parking points of observable targets, and establishing a set of observation options including multiple perspectives. Output a set of compensatory observation options to the user and capture the ideal observation point actually selected by the user; extract the spatial offset parameters between the ideal observation point and the initially expected target movement path; perform intent feedback parsing based on the spatial offset parameters and organize the intent feedback features. Extract the residual tension value of intent from the intent deviation feedback features, analyze the number of observation compensation nodes, and generate a micro-cruise trajectory around the target along the outer contour of the physical safety boundary; perform multi-view alternating observations on the micro-cruise trajectory to establish an array of observation confirmation points.

2. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 1, characterized in that: The security-intent conflict identifier is established as follows: By retrieving the equipment geographic information system ledger and static topology map, a spatial index grid for the robot is established; Obtain the spatial trajectory coordinate sequence of the target's movement path, compare and calculate the spatial position of each coordinate point in the spatial trajectory coordinate sequence with the three-dimensional safety protection surface, and determine whether the target's movement path crosses the physical safety boundary and intrudes into the internal space on the side of the hazard source. If the judgment result is that there is an intrusion, then it is determined that all continuous coordinate points located inside the physical security boundary are extracted sequentially from the spatial trajectory coordinate sequence, and the segments that are fitted to overlap in space are extracted as the trajectory overlap segments. Extract the start coordinates, end coordinates, and intrusion depth parameters of the overlapping segments, generate an identifier field, and define it as a security-intent conflict identifier.

3. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 1, characterized in that: The method for filtering the candidate stopping points is as follows: By scanning the field of view around the current location using a power inspection robot, the surrounding terrain data and the coordinates of static obstacles outside the physical safety boundary can be obtained. Based on the surrounding terrain data and the coordinates of static obstacle distribution, multiple safe-passing surrounding physical grids are delineated as alternative areas in the spatial index grid structure. For each candidate area, obtain the spatial coordinates within the candidate area and analyze the line-of-sight connectivity from the spatial coordinates to the abnormal area of ​​the target device; Spatial coordinate points with expected line-of-sight connectivity are extracted through spatial clustering to form candidate stopping points for effective visual coverage.

4. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 3, characterized in that: The method for analyzing the line-of-sight connectivity is as follows: Define the target equipment observation surface for the abnormal area of ​​the target equipment, and construct a spatial straight line connecting the spatial coordinate points of the candidate area and the target equipment observation surface in the three-dimensional coordinate system; If the spatial straight line does not intersect or cut off any entity structure coordinates in the static topology diagram, then the line-of-sight connectivity is determined to be as expected.

5. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 1, characterized in that: The method for generating the compensatory observation option set is as follows: Extract the predicted observation angle, the relative observation distance to the target device, and the spatial angle between the predicted observation angle and the frontal normal vector of the target device corresponding to all candidate parking points. Establish a comprehensive evaluation rule, sort all candidate parking points, and obtain a sorted list. The candidate stopping points in the sorted list are truncated, and their spatial location parameters and viewpoint features are extracted and combined to form a compensatory observation option set.

6. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 1, characterized in that: The method for performing the intent feedback parsing is as follows: Extract the spatial concession coefficient for the degree of physical retreat; The peak sound pressure level of the secondary voice signal when the user issues the final command is called, and the ratio of the difference between the peak sound pressure level and the peak sound pressure level of the original stressed voice command is calculated to obtain the stress attenuation rate. Intent residual tension is assessed based on the original stress speech sound pressure peak value and spatial concession coefficient to obtain residual tension value; The intended residual tension value is integrated with the actual observation point coordinates to serve as the intended deviation feedback feature.

7. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 6, characterized in that: The method for performing the intentional residual tension assessment is as follows: The peak sound pressure level of the original stressed speech was extracted as the initial intention potential energy, and the spatial concession coefficient and stress attenuation rate were respectively used as two characteristic parameters affecting tension release. By combining two characteristic influencing parameters to perform attenuation analysis on the initial intention potential energy, the residual tension value characterizing the user's internal resistance intensity is obtained.

8. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 1, characterized in that: The micro-cruising trajectory is generated as follows: Based on the intention deviation feedback characteristics, a node tension multiplication analysis was performed to obtain the total number of observed compensation nodes; With the center coordinates of the abnormal area of ​​the target equipment as the center and the safety clearance distance corresponding to the physical safety boundary as the radius, construct a safety arc facing the abnormal area of ​​the target equipment. Extract the projected coordinates of the actual selected observation point on the safety arc, and use the projected coordinates as the starting anchor point of the micro-cruise trajectory; Based on the starting anchor point, and according to the pre-set fixed included angle step size, the system alternately and symmetrically expands to both sides along the safety arc, and sequentially intercepts a number of spatial coordinate points equal to the total number of observation compensation nodes, forming a distributed observation node array. All spatial coordinate points in the distributed observation node array are smoothly connected sequentially according to their spatial proximity to generate a micro-cruise trajectory around the abnormal area of ​​the target device.

9. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 1, characterized in that: By extracting the residual tension value of intent from the intent deviation feedback features and the distributed observation node array, and combining it with the physical safety boundary, key observation trigger areas are identified; at the same time, a dynamic control strategy for autonomous inspection within the key observation trigger areas is established.

10. The interaction method for an energy storage power inspection robot based on multimodal semantic understanding according to claim 9, characterized in that: The method for identifying the key observation trigger area is as follows: Extract the residual tension value of intent stored in the intent deviation feedback feature, and retrieve the three-dimensional coordinate set of all final observation confirmation points contained in the distributed observation node array; In the spatial index grid, based on the physical contour formed by the set of three-dimensional coordinates and the micro-cruise trajectory, an outward spatial three-dimensional grid expansion operation is performed. The intentional residual tension value is rounded down and converted into the number of mesh expansion layers for the spatial three-dimensional mesh expansion operation; After completing the spatial three-dimensional mesh expansion operation based on the number of mesh expansion layers as the expansion distance, a three-dimensional spatial mesh block is generated outside the physical safety boundary of the abnormal area of ​​the target device, which is defined as the key observation trigger area.

Images