Intelligent fire-fighting man-machine interaction method and system and computer readable storage medium
Through the visual perception matching, virtual reality technology and voice recognition of intelligent fire protection systems, combined with eye movement and head movement data, the user interface and interaction methods in traditional fire protection systems have been upgraded, solving the problem of single interaction methods of traditional fire protection systems, and improving the problem of Operational efficiency and interactivity.
Patent Information
- Application Number
- CN202411110350.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The user interface and interaction methods of traditional fire protection systems are relatively old-fashioned and single, and cannot provide a personalized experience or interactive operation experience, and the feedback is not detailed enough, making it difficult to meet the complex needs of users.
Using the human-computer interaction method of intelligent firefighting, the real-time video interface is obtained through the intelligent firefighting robot to match the visual perception intensity with the user, and virtual reality technology is used to combine eye movement and head movement data to perform interactive operations; user instructions are identified through the voice recognition model. And perform emotional extraction and matching commands to generate intelligent control commands.
Improves the user's visual experience and operation efficiency in the virtual reality environment, realizes a more natural and intuitive interaction method, enhances the interaction and interaction accuracy between users and intelligent firefighting robots, and can accurately understand and execute user intentions. Provide more humanized services.
Smart Images

Figure CN119024965A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method and system for intelligent firefighting, and a computer-readable storage medium. Background Art
[0002] Human-computer interaction refers to the interaction and communication process between humans and computer systems. This interaction can cover a variety of forms and technologies, aiming to enable humans to effectively use and control computer systems, and computer systems to respond to and meet user needs in a user-friendly manner; intelligent fire protection is a modern fire protection that uses advanced information technology and intelligent equipment, combined with the actual needs of fire protection engineering, to improve fire safety, emergency response efficiency and management level.
[0003] Traditional methods usually rely on physical buttons, switches or complex control panels for operation, which may require training or professional skills to perform correctly; and the feedback of traditional methods is usually limited to simple indicator lights or sound alarms, which cannot provide detailed scene information or confirmation of user intentions; and the user interface and interaction method of traditional methods are relatively old-fashioned and single, and usually cannot provide personalized experience or interactive operation experience. Summary of the invention
[0004] The purpose of the present invention is to address the problems existing in the background technology and to propose a human-computer interaction method, system and computer-readable storage medium for intelligent firefighting.
[0005] The technical solution of the present invention is a human-computer interaction method for intelligent fire fighting, comprising:
[0006] Based on the intelligent fire-fighting robot, a real-time video interface of the target fire-fighting area is obtained, and the real-time video interface and the target operating user are matched with visual perception intensity to obtain a real-time video interface adapted to the target operating user;
[0007] Based on virtual reality technology, a real-time video interface adapted to a target operating user is mapped to a VR device, and eye movement data and head movement data of the target operating user are collected based on the VR device to obtain eye movement control data and head movement control data of the target operating user;
[0008] Performing interactive operations on the real-time video interface based on the eye movement control data and head movement control data of the target operation user;
[0009] The target operating user issues a control voice to the intelligent firefighting robot based on the real-time video interface in the VR device;
[0010] Recognize the control speech through a trained speech recognition model to obtain a control text, and perform sentiment extraction on the control text to obtain a sentiment score corresponding to the control text;
[0011] The control text and the emotion score corresponding to the control text are matched with a preset control command table to obtain the control command of the intelligent fire-fighting robot.
[0012] Preferably, matching the visual perception intensity of the real-time video interface and the target operating user to obtain a real-time video interface adapted to the target operating user includes:
[0013] The real-time video interface is divided into a plurality of real-time video sub-interfaces, and the plurality of real-time video sub-interfaces are encoded, and the encoding sequence numbers of the plurality of real-time video sub-interfaces are as follows:
[0014] H = {1,,2,...,h};
[0015] Wherein, H represents the total set of coding sequence numbers, and h represents the number of real-time video sub-interfaces in the real-time video interface;
[0016] pre-defining the importance, area and intensity levels of the visual perception elements of the plurality of real-time video sub-interfaces;
[0017] The visual perception intensity of the real-time video interface is calculated based on the importance, area and intensity level of the visual perception elements of the multiple real-time video sub-interfaces. The visual perception intensity calculation formula of the real-time video interface is as follows:
[0018]
[0019] Where O represents the visual perception intensity of the real-time video interface, s j represents the importance of the visual perception element of the jth real-time video sub-interface, r j represents the area of the visual perception element of the jth real-time video sub-interface, A j represents the intensity level of the visual perception element of the j-th real-time video sub-interface;
[0020] An objective function is constructed based on the visual perception intensity of the real-time video interface, a constraint condition is constructed based on the areas of the multiple real-time video sub-interfaces, and a matching optimization model is constructed based on the objective function and the constraint condition. The matching optimization model is as follows:
[0021]
[0022] Among them, max represents the maximum value function, st represents the constraint condition, R j represents the area of the jth real-time video sub-interface;
[0023] The matching optimization model is solved based on an optimization algorithm to obtain a real-time video interface that is suitable for a target operating user.
[0024] Preferably, solving the matching optimization model based on an optimization algorithm to obtain a real-time video interface adapted to the target operating user includes:
[0025] Initializing an iteration counter, a population size, and a maximum number of iterations of the optimization algorithm, and randomly generating an initial population, mapping the importance, area, and intensity level of the visual perception elements of the matching optimization model to individual positions of the optimization algorithm;
[0026] The objective function value of each individual in the population is calculated, and the position of the individual is updated. The position update formula is as follows:
[0027]
[0028] Among them, X i (t+1) and X i (t) represents the individual position at the t+1th and tth iterations, represents the existing optimal solution, r represents the spiral constant, and z represents a random value in the interval [0,1];
[0029] The new population is evaluated and calculated to find the global optimal individual and individual position, and the optimal solution is updated. It is determined whether the number of iterations of the optimization algorithm reaches the preset maximum number of iterations. If the preset maximum number of iterations is reached, the optimal solution is output, otherwise, the above steps are repeated.
[0030] Preferably, performing interactive operation on the real-time video interface based on the eye movement control data and head movement control data of the target operation user includes:
[0031] Fitting the eye movement control data of the target operating user through an eye movement polynomial to obtain the eye gaze coordinates of the target operating user in the real-time video interface;
[0032] Based on the head movement control data of the target operation user, it is determined whether the eye gaze coordinates are within a preset range. If they are within the preset range, the eye gaze coordinates are used as the eye gaze target. Otherwise, the above steps are repeated;
[0033] The spatial head gaze coordinates of the target operating user in the real-time video interface are calculated based on the head movement control data of the target operating user, and the angle between the spatial head gaze coordinates and the target operating user is calculated.
[0034] Preferably, performing interactive operation on the real-time video interface based on the eye movement control data and head movement control data of the target operation user further includes:
[0035] Determine whether the angle between the spatial head gaze coordinates and the target operating user is less than a preset angle threshold, and if so, add the spatial head gaze coordinates to an intention set;
[0036] Sorting the spatial head gaze coordinates in time series to obtain a spatial head gaze coordinate sequence, and calculating a change speed of the spatial head gaze coordinates based on the spatial head gaze coordinate sequence;
[0037] Determine whether the change speed of the spatial head gaze coordinates is less than a preset change speed threshold. If so, determine whether the glasses gaze target is in the intention set. If so, perform interactive operations based on the glasses gaze target. Otherwise, repeat the above steps.
[0038] Preferably, the eye movement polynomial is as follows:
[0039]
[0040] Among them, (x s ,y s ) represents the coordinates of the eye gaze point on the two-dimensional real-time video interface, a0, a1, a2, a3, a4 and a5 represent the first polynomial regression coefficients, b0, b1, b2, b3, b4 and b5 represent the second polynomial regression coefficients, V x and V y It is expressed as the horizontal and vertical offset between the pupil center and the midpoint of the light spot;
[0041] The calculation formula of the spatial head gaze coordinates of the target operation user in the real-time video interface is as follows:
[0042]
[0043] Where (x, y, z) represents the spatial head gaze coordinates of the target operating user in the real-time video interface, (e1, e2, e3) represents the starting point coordinates of the head gaze ray in the head movement control data, (s1, s2, s3) represents the coordinates of the direction vector of the head gaze ray in the head movement control data, j represents a temporary variable, and n i Represents the normal vector of the real-time video interface, b i Indicates the world coordinates of the target operation user;
[0044] The angle calculation formula between the spatial head gaze coordinates and the target operating user is as follows:
[0045]
[0046] Among them, θ represents the angle between the spatial head gaze coordinates and the target operating user, S represents the direction vector of the head gaze ray, M represents the direction vector between the starting point coordinates of the head gaze ray in the head movement control data and the world coordinates of the target operating user, and SM represents the vector between the direction vector of the head gaze ray and the direction vector between the starting point coordinates of the head gaze ray in the head movement control data and the world coordinates of the target operating user.
[0047] Preferably, performing sentiment extraction on the control text to obtain a sentiment score corresponding to the control text includes:
[0048] Pre-constructing a sentiment dictionary, wherein the sentiment dictionary includes a target sentiment word, a sentiment polarity of the target sentiment word, and a sentiment intensity of the target sentiment word;
[0049] Matching the control text with the sentiment dictionary to obtain the target sentiment word in the control text, and obtaining the word segment before the target sentiment word in the control text;
[0050] Matching the word segment before the target sentiment word in the control text with a preset degree adverb table, if the word segment matches the degree adverb table, acquiring the sentiment intensity of the word segment based on the sentiment dictionary, and calculating the first sentiment score of the target sentiment word based on the sentiment intensity of the word segment, otherwise, calculating the second sentiment score of the target sentiment word based on the type of the word segment;
[0051] The sentiment score corresponding to the control text is acquired based on the first sentiment score and the second sentiment score of each target sentiment word in the control text.
[0052] Preferably, the first sentiment score calculation formula of the target sentiment word is as follows:
[0053] S i =p wi ·s wi ·d i ;
[0054] Among them, S i represents the first sentiment score of the target sentiment word, p wi Indicates the sentiment polarity of the target sentiment word, s wi Indicates the sentiment intensity of the target sentiment word, d i Indicates the emotional intensity of the word segment;
[0055] The second sentiment score calculation formula of the target sentiment word is as follows:
[0056]
[0057] Among them, S i ′ represents the second sentiment score of the target sentiment word. When the type of the word segment is a negative word, n i =1, when the type of the word segment is a negative word, n i =1, when the type of the word segment is not a negative word, n i =0.
[0058] The technical solution of the present invention is a human-computer interaction system for intelligent firefighting, which is applicable to the human-computer interaction method for intelligent firefighting, comprising:
[0059] An interface acquisition module, which is used to acquire a real-time video interface of a target fire fighting area based on the intelligent fire fighting robot;
[0060] An intensity matching module, the intensity matching module is used to match the visual perception intensity of the real-time video interface and the target operating user to obtain a real-time video interface adapted to the target operating user, and map the real-time video interface adapted to the target operating user to the VR device based on virtual reality technology;
[0061] A data acquisition module, wherein the data acquisition module is used to collect eye movement data and head movement data of the target operation user based on the VR device to obtain eye movement control data and head movement control data of the target operation user;
[0062] A control recognition module, wherein the control recognition module is used to recognize the control voice through a trained voice recognition model to obtain a control text;
[0063] A sentiment analysis module, wherein the sentiment analysis module is used to extract sentiment from the control text to obtain a sentiment score corresponding to the control text;
[0064] A command matching module is used to match the control text and the emotion score corresponding to the control text with a preset control command table to obtain the control command of the intelligent fire-fighting robot.
[0065] The technical solution of the present invention is: a computer-readable storage medium, which is suitable for the human-computer interaction method of intelligent firefighting. The computer-readable storage medium stores computer execution instructions, and the computer execution instructions are used to implement the human-computer interaction method of intelligent firefighting when executed by a processor.
[0066] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:
[0067] 1. The present invention can ensure that the user obtains the best visual experience in the virtual reality environment by matching the real-time video interface with the visual perception intensity of the target operating user. This matching can be adjusted according to factors such as the user's visual habits and vision conditions to improve the user's operating efficiency and comfort. The VR device is used to collect the user's eye movement control data and head movement control data to achieve a more natural and intuitive interaction method. This method not only enhances the interactivity between the user and the intelligent fire-fighting robot, but also effectively captures the user's intentions and actions, and improves the accuracy and efficiency of the interaction.
[0068] 2. The present invention controls the intelligent fire-fighting robot by issuing voice commands through VR devices. This method greatly enhances the user's operating convenience and real-time response capability. Through a trained voice recognition model, it can accurately recognize the user's voice commands, thereby executing corresponding control tasks, and extract emotions from the control text and generate emotional scores. The robot's operational response can be adjusted according to the user's emotional state. This intelligent emotion recognition capability enables the intelligent fire-fighting system to better understand and adapt to the user's emotional changes, provide more humane services and support, and match the control text with a preset control command table to ensure that the intelligent fire-fighting robot can accurately understand and execute the user's commands. This matching process ensures the stability and reliability of the system, allowing users to easily operate the fire-fighting robot through voice and visual interaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 A schematic flow chart of an overall method in an embodiment of the present invention;
[0070] Figure 2 A schematic diagram of the overall system flow in an embodiment of the present invention.
[0071] Figure numerals: 1. Interface acquisition module; 2. Intensity matching module; 3. Data acquisition module; 4. Control recognition module; 5. Sentiment analysis module; 6. Command matching module. DETAILED DESCRIPTION
[0072] Embodiment 1, as Figure 1 As shown, the present invention proposes a human-computer interaction method for intelligent firefighting, comprising:
[0073] S1. Based on the intelligent firefighting robot, a real-time video interface of the target firefighting area is obtained, and the real-time video interface and the target operating user are matched with visual perception intensity to obtain a real-time video interface suitable for the target operating user;
[0074] S2. Mapping a real-time video interface adapted to the target operating user to a VR device based on virtual reality technology, and collecting eye movement data and head movement data of the target operating user based on the VR device to obtain eye movement control data and head movement control data of the target operating user;
[0075] S3, performing interactive operations on the real-time video interface based on the eye movement control data and head movement control data of the target operation user;
[0076] S4, the target operation user issues a control voice for the intelligent firefighting robot based on the real-time video interface in the VR device;
[0077] S5. Recognize the control speech through the trained speech recognition model to obtain the control text, and extract the emotion of the control text to obtain the emotion score corresponding to the control text;
[0078] S6. Match the control text and the emotion score corresponding to the control text with a preset control command table to obtain the control command of the intelligent fire-fighting robot.
[0079] In the present invention, the real-time video interface is that the intelligent fire-fighting robot obtains real-time video of the target fire-fighting area and displays the current fire-fighting situation in real time; virtual reality technology is the use of virtual reality (VR) equipment to display the video interface of the fire scene in real time through devices such as head-mounted displays, and let the user feel that he is at the scene; the control voice is that the user controls the behavior of the intelligent fire-fighting robot or performs a specific task through the voice commands spoken in the VR device; the speech recognition model is a trained artificial intelligence model, which is used to convert the voice commands issued by the user into text form; the control command table is a pre-set command set, which includes various possible control commands and corresponding operations, and is used to determine the final operation command according to the user's voice command and emotional score.
[0080] Embodiment 2, a human-computer interaction method for intelligent firefighting proposed by the present invention, compared with embodiment 1, this embodiment further includes: matching the visual perception intensity of the real-time video interface and the target operating user to obtain a real-time video interface adapted to the target operating user, including:
[0081] A1. Divide the real-time video interface into multiple real-time video sub-interfaces, and encode the multiple real-time video sub-interfaces. The encoding sequence numbers of the multiple real-time video sub-interfaces are as follows:
[0082] H = {1,,2,...,h};
[0083] Wherein, H represents the total set of coding sequence numbers, and h represents the number of real-time video sub-interfaces in the real-time video interface;
[0084] A2. pre-define the importance, area and intensity level of visual perception elements of multiple real-time video sub-interfaces;
[0085] A3. The visual perception intensity of the real-time video interface is calculated based on the importance, area and intensity level of the visual perception elements of multiple real-time video sub-interfaces. The visual perception intensity calculation formula of the real-time video interface is as follows:
[0086]
[0087] Where O represents the visual perception intensity of the real-time video interface, s j represents the importance of the visual perception element of the jth real-time video sub-interface, r j represents the area of the visual perception element of the jth real-time video sub-interface, A j represents the intensity level of the visual perception element of the j-th real-time video sub-interface;
[0088] A4. Construct an objective function based on the visual perception intensity of the real-time video interface, construct constraint conditions based on the areas of multiple real-time video sub-interfaces, and construct a matching optimization model based on the objective function and constraint conditions. The matching optimization model is as follows:
[0089]
[0090] Among them, max represents the maximum value function, st represents the constraint condition, R j represents the area of the jth real-time video sub-interface;
[0091] A5. Solve the matching optimization model based on the optimization algorithm to obtain a real-time video interface that is suitable for the target operation user.
[0092] In an optional embodiment, solving the matching optimization model based on an optimization algorithm to obtain a real-time video interface adapted to a target operating user includes:
[0093] B1, initializing the iteration counter, population size and maximum number of iterations of the optimization algorithm, and randomly generating an initial population, mapping the importance, area and intensity level of the visual perception elements matching the optimization model to the individual positions of the optimization algorithm;
[0094] B2. Calculate the objective function value of each individual in the population and update the position of the individual. The position update formula is as follows:
[0095]
[0096] Among them, X i (t+1) and X i (t) represents the individual position at the t+1th and tth iterations, represents the existing optimal solution, r represents the spiral constant, and z represents a random value in the interval [0,1];
[0097] B3. Perform computational evaluation on the new population to find the global optimal individual and individual position, and update the optimal solution. Determine whether the number of iterations of the optimization algorithm reaches the preset maximum number of iterations. If it reaches the preset maximum number of iterations, output the optimal solution. Otherwise, repeat the above steps.
[0098] In an optional embodiment, performing interactive operations on a real-time video interface based on eye movement control data and head movement control data of a target operation user includes:
[0099] D1. Fitting the eye movement control data of the target operating user through an eye movement polynomial to obtain the eye gaze coordinates of the target operating user in the real-time video interface;
[0100] D2. judging whether the eye gaze coordinates are within a preset range based on the head movement control data of the target operation user; if so, taking the eye gaze coordinates as the eye gaze target; otherwise, repeating the above steps;
[0101] D3. Calculate the spatial head gaze coordinates of the target operating user in the real-time video interface based on the head movement control data of the target operating user, and calculate the angle between the spatial head gaze coordinates and the target operating user.
[0102] It should be noted that eye control data is data obtained through eye tracking devices, which records the position and duration of the user's eye gaze when watching a video; eye movement polynomial refers to a method of fitting eye movement data with a polynomial function, which is usually used to smooth and simulate eye movement trajectories to predict the user's gaze point; eye gaze coordinates are the specific position coordinates of the user's eyes on the video interface calculated based on eye movement control data; head movement control data is data obtained through head tracking devices, which records the position and posture changes of the user's head in space; spatial head gaze coordinates are the gaze position coordinates of the user's head on the video interface calculated based on the head movement control data.
[0103] In an optional embodiment, the interactive operation on the real-time video interface based on the eye movement control data and the head movement control data of the target operation user further includes:
[0104] D4, determining whether the angle between the spatial head gaze coordinates and the target operating user is less than a preset angle threshold. If so, adding the spatial head gaze coordinates to the intention set;
[0105] D5, sorting the spatial head gaze coordinates in time series to obtain a spatial head gaze coordinate sequence, and calculating a change speed of the spatial head gaze coordinates based on the spatial head gaze coordinate sequence;
[0106] D6. Determine whether the change speed of the spatial head gaze coordinates is less than the preset change speed threshold. If it is less than the preset change speed threshold, determine whether the glasses gaze target is in the intention set. If it is in the intention set, perform interactive operations based on the glasses gaze target. Otherwise, repeat the above steps.
[0107] It should be noted that the angle refers to the angle between two vectors or coordinates in space, which is used to describe the relative position between the head gaze direction and the target operation user; the intention set is a set that records the targets or operations that the user may be interested in, usually judging the user's intention through gaze coordinates or action data; change speed: refers to the rate of change of the head gaze coordinates over time, which is used to analyze the movement state of the user's head.
[0108] In an optional embodiment, the eye movement polynomial is as follows:
[0109]
[0110] Among them, (x s ,y s ) represents the coordinates of the eye gaze point on the two-dimensional real-time video interface, a0, a1, a2, a3, a4 and a5 represent the first polynomial regression coefficients, b0, b1, b2, b3, b4 and b5 represent the second polynomial regression coefficients, V x and V y It is expressed as the horizontal and vertical offset between the pupil center and the midpoint of the light spot;
[0111] The calculation formula for the spatial head gaze coordinates of the target operation user in the real-time video interface is as follows:
[0112]
[0113] Where (x, y, z) represents the spatial head gaze coordinates of the target operating user in the real-time video interface, (e1, e2, e3) represents the starting point coordinates of the head gaze ray in the head movement control data, (s1, s2, s3) represents the coordinates of the direction vector of the head gaze ray in the head movement control data, j represents a temporary variable, and n i Represents the normal vector of the real-time video interface, b i Indicates the world coordinates of the target operation user;
[0114] The angle calculation formula between the spatial head gaze coordinates and the target operation user is as follows:
[0115]
[0116] Among them, θ represents the angle between the spatial head gaze coordinates and the target operating user, S represents the direction vector of the head gaze ray, M represents the direction vector between the starting point coordinates of the head gaze ray in the head movement control data and the world coordinates of the target operating user, and SM represents the vector between the direction vector of the head gaze ray and the direction vector between the starting point coordinates of the head gaze ray in the head movement control data and the world coordinates of the target operating user.
[0117] In an optional embodiment, sentiment extraction is performed on the control text to obtain a sentiment score corresponding to the control text, including:
[0118] E1. Pre-build a sentiment dictionary, which includes target sentiment words, sentiment polarity of target sentiment words, and sentiment intensity of target sentiment words;
[0119] E2, matching the control text with the sentiment dictionary to obtain the target sentiment word in the control text, and obtaining the word segment before the target sentiment word in the control text;
[0120] E3, matching the word segment before the target sentiment word in the control text with the preset degree adverb table, if the word segment matches the degree adverb table, obtaining the sentiment intensity of the word segment based on the sentiment dictionary, and calculating the first sentiment score of the target sentiment word based on the sentiment intensity of the word segment, otherwise, calculating the second sentiment score of the target sentiment word based on the type of the word segment;
[0121] E4. Obtain the sentiment score corresponding to the control text based on the first sentiment score and the second sentiment score of each target sentiment word in the control text.
[0122] In an optional embodiment, the first sentiment score calculation formula of the target sentiment word is as follows:
[0123] S i =p wi ·s wi ·d i ;
[0124] Among them, S i represents the first sentiment score of the target sentiment word, p wi Indicates the sentiment polarity of the target sentiment word, s wi Indicates the sentiment intensity of the target sentiment word, d i Indicates the emotional intensity of the word segment;
[0125] The calculation formula for the second sentiment score of the target sentiment word is as follows:
[0126]
[0127] Among them, S i ′ represents the second sentiment score of the target sentiment word. When the type of the word segment is a negative word, n i =1, when the type of the word segment is a negative word, n i =1, when the type of the word segment is not a negative word, n i =0.
[0128] Embodiment three, as Figure 2 As shown, the human-computer interaction system for intelligent fire fighting proposed by the present invention is applicable to the human-computer interaction method for intelligent fire fighting, and comprises:
[0129] Interface acquisition module 1, the interface acquisition module 1 is used to obtain a real-time video interface of a target fire fighting area based on the intelligent fire fighting robot;
[0130] The intensity matching module 2 is used to match the visual perception intensity of the real-time video interface and the target operating user to obtain a real-time video interface adapted to the target operating user, and map the real-time video interface adapted to the target operating user to the VR device based on virtual reality technology;
[0131] Data collection module 3, the data collection module 3 is used to collect the eye movement data and head movement data of the target operation user based on the VR device to obtain the eye movement control data and head movement control data of the target operation user;
[0132] A control recognition module 4, which is used to recognize the control voice through a trained voice recognition model to obtain a control text;
[0133] Sentiment analysis module 5, which is used to extract sentiment from the control text to obtain a sentiment score corresponding to the control text;
[0134] The command matching module 6 is used to match the control text and the emotion score corresponding to the control text with a preset control command table to obtain the control command of the intelligent fire-fighting robot.
[0135] In the fourth embodiment, a computer-readable storage medium proposed in the present invention is applicable to a human-computer interaction method for intelligent firefighting. The computer-readable storage medium stores computer execution instructions, and the computer execution instructions are used to implement a human-computer interaction method for intelligent firefighting when executed by a processor.
[0136] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.
Claims
1. A human-computer interaction method for intelligent firefighting, characterized in that: include: Based on the intelligent fire-fighting robot, a real-time video interface of the target fire-fighting area is obtained, and the real-time video interface and the target operating user are matched with visual perception intensity to obtain a real-time video interface adapted to the target operating user; Based on virtual reality technology, a real-time video interface adapted to a target operating user is mapped to a VR device, and eye movement data and head movement data of the target operating user are collected based on the VR device to obtain eye movement control data and head movement control data of the target operating user; Performing interactive operations on the real-time video interface based on the eye movement control data and head movement control data of the target operation user; The target operating user issues a control voice to the intelligent firefighting robot based on the real-time video interface in the VR device; Recognize the control speech through a trained speech recognition model to obtain a control text, and perform sentiment extraction on the control text to obtain a sentiment score corresponding to the control text; The control text and the emotion score corresponding to the control text are matched with a preset control command table to obtain the control command of the intelligent fire-fighting robot.
2. The human-computer interaction method for intelligent firefighting according to claim 1, characterized in that: Matching the visual perception intensity of the real-time video interface and the target operating user to obtain a real-time video interface adapted to the target operating user includes: The real-time video interface is divided into a plurality of real-time video sub-interfaces, and the plurality of real-time video sub-interfaces are encoded, and the encoding sequence numbers of the plurality of real-time video sub-interfaces are as follows: H = {1,,2,...,h}; Wherein, H represents the total set of coding sequence numbers, and h represents the number of real-time video sub-interfaces in the real-time video interface; pre-defining the importance, area and intensity levels of the visual perception elements of the plurality of real-time video sub-interfaces; The visual perception intensity of the real-time video interface is calculated based on the importance, area and intensity level of the visual perception elements of the multiple real-time video sub-interfaces. The visual perception intensity calculation formula of the real-time video interface is as follows: Where O represents the visual perception intensity of the real-time video interface, s j represents the importance of the visual perception element of the jth real-time video sub-interface, r j represents the area of the visual perception element of the jth real-time video sub-interface, A j represents the intensity level of the visual perception element of the j-th real-time video sub-interface; An objective function is constructed based on the visual perception intensity of the real-time video interface, a constraint condition is constructed based on the areas of the multiple real-time video sub-interfaces, and a matching optimization model is constructed based on the objective function and the constraint condition. The matching optimization model is as follows: Among them, max represents the maximum value function, st represents the constraint condition, R j represents the area of the jth real-time video sub-interface; The matching optimization model is solved based on an optimization algorithm to obtain a real-time video interface that is suitable for a target operating user.
3. The human-computer interaction method for intelligent firefighting according to claim 2, characterized in that: Solving the matching optimization model based on an optimization algorithm to obtain a real-time video interface adapted to a target operating user includes: Initializing an iteration counter, a population size, and a maximum number of iterations of the optimization algorithm, and randomly generating an initial population, mapping the importance, area, and intensity level of the visual perception elements of the matching optimization model to individual positions of the optimization algorithm; The objective function value of each individual in the population is calculated, and the position of the individual is updated. The position update formula is as follows: Among them, X i (t+1) and X i (t) represents the individual position at the t+1th and tth iterations, represents the existing optimal solution, r represents the spiral constant, and z represents a random value in the interval [0,1]; The new population is evaluated and calculated to find the global optimal individual and individual position, and the optimal solution is updated. It is determined whether the number of iterations of the optimization algorithm reaches the preset maximum number of iterations. If the preset maximum number of iterations is reached, the optimal solution is output, otherwise, the above steps are repeated.
4. The human-computer interaction method for intelligent firefighting according to claim 1, characterized in that: Performing interactive operations on the real-time video interface based on the eye movement control data and the head movement control data of the target operation user includes: Fitting the eye movement control data of the target operating user through an eye movement polynomial to obtain the eye gaze coordinates of the target operating user in the real-time video interface; Based on the head movement control data of the target operation user, it is determined whether the eye gaze coordinates are within a preset range. If they are within the preset range, the eye gaze coordinates are used as the eye gaze target. Otherwise, the above steps are repeated; The spatial head gaze coordinates of the target operating user in the real-time video interface are calculated based on the head movement control data of the target operating user, and the angle between the spatial head gaze coordinates and the target operating user is calculated.
5. The human-computer interaction method for intelligent firefighting according to claim 4, characterized in that: The real-time video interface is interactively operated based on the eye movement control data and the head movement control data of the target operation user, further comprising: Determine whether the angle between the spatial head gaze coordinates and the target operating user is less than a preset angle threshold, and if so, add the spatial head gaze coordinates to an intention set; Sorting the spatial head gaze coordinates in time series to obtain a spatial head gaze coordinate sequence, and calculating a change speed of the spatial head gaze coordinates based on the spatial head gaze coordinate sequence; Determine whether the change speed of the spatial head gaze coordinates is less than a preset change speed threshold. If so, determine whether the glasses gaze target is in the intention set. If so, perform interactive operations based on the glasses gaze target. Otherwise, repeat the above steps.
6. The human-computer interaction method for intelligent firefighting according to claim 5, characterized in that: The eye movement polynomial is as follows: Among them, (x s ,y s ) represents the coordinates of the eye gaze point on the two-dimensional real-time video interface, a0, a1, a2, a3, a4 and a5 represent the first polynomial regression coefficients, b0, b1, b2, b3, b4 and b5 represent the second polynomial regression coefficients, V x and V y It is expressed as the horizontal and vertical offset between the pupil center and the midpoint of the light spot; The calculation formula of the spatial head gaze coordinates of the target operation user in the real-time video interface is as follows: Where (x, y, z) represents the spatial head gaze coordinates of the target operating user in the real-time video interface, (e1, e2, e3) represents the starting point coordinates of the head gaze ray in the head movement control data, (s1, s2, s3) represents the coordinates of the direction vector of the head gaze ray in the head movement control data, j represents a temporary variable, and n i Represents the normal vector of the real-time video interface, b i Indicates the world coordinates of the target operation user; The angle calculation formula between the spatial head gaze coordinates and the target operating user is as follows: Among them, θ represents the angle between the spatial head gaze coordinates and the target operating user, S represents the direction vector of the head gaze ray, M represents the direction vector between the starting point coordinates of the head gaze ray in the head movement control data and the world coordinates of the target operating user, and SM represents the vector between the direction vector of the head gaze ray and the direction vector between the starting point coordinates of the head gaze ray in the head movement control data and the world coordinates of the target operating user.
7. The human-computer interaction method for intelligent firefighting according to claim 1, characterized in that: Performing sentiment extraction on the control text to obtain a sentiment score corresponding to the control text includes: Pre-constructing a sentiment dictionary, wherein the sentiment dictionary includes a target sentiment word, a sentiment polarity of the target sentiment word, and a sentiment intensity of the target sentiment word; Matching the control text with the sentiment dictionary to obtain the target sentiment word in the control text, and obtaining the word segment before the target sentiment word in the control text; Matching the word segment before the target sentiment word in the control text with a preset degree adverb table, if the word segment matches the degree adverb table, acquiring the sentiment intensity of the word segment based on the sentiment dictionary, and calculating the first sentiment score of the target sentiment word based on the sentiment intensity of the word segment, otherwise, calculating the second sentiment score of the target sentiment word based on the type of the word segment; The sentiment score corresponding to the control text is acquired based on the first sentiment score and the second sentiment score of each target sentiment word in the control text.
8. The human-computer interaction method for intelligent firefighting according to claim 7, characterized in that: The calculation formula for the first sentiment score of the target sentiment word is as follows: S i =p wi ·s wi ·d i ; Among them, S i represents the first sentiment score of the target sentiment word, p wi Indicates the sentiment polarity of the target sentiment word, s wi Indicates the sentiment intensity of the target sentiment word, d i Indicates the emotional intensity of the word segment; The second sentiment score calculation formula of the target sentiment word is as follows: Among them, S i ′ represents the second sentiment score of the target sentiment word. When the type of the word segment is a negative word, n i =1, when the type of the word segment is a negative word, n i =1, when the type of the word segment is not a negative word, n i =0.
9. A human-computer interaction system for intelligent firefighting, which is applicable to a human-computer interaction method for intelligent firefighting according to any one of claims 1 to 8, characterized in that: include: An interface acquisition module (1), the interface acquisition module (1) being used to acquire a real-time video interface of a target fire fighting area based on the intelligent fire fighting robot; An intensity matching module (2), the intensity matching module (2) is used to match the visual perception intensity of the real-time video interface and the target operating user to obtain a real-time video interface suitable for the target operating user, and map the real-time video interface suitable for the target operating user to a VR device based on virtual reality technology; A data acquisition module (3), the data acquisition module (3) being used to acquire eye movement data and head movement data of the target operating user based on the VR device, so as to obtain eye movement control data and head movement control data of the target operating user; A control recognition module (4), the control recognition module (4) is used to recognize the control voice through a trained voice recognition model to obtain a control text; A sentiment analysis module (5), wherein the sentiment analysis module (5) is used to extract sentiment from the control text to obtain a sentiment score corresponding to the control text; A command matching module (6), the command matching module (6) is used to match the control text and the emotion score corresponding to the control text with a preset control command table to obtain the control command of the intelligent fire-fighting robot.
10. A computer-readable storage medium is applicable to the human-computer interaction method for intelligent firefighting according to any one of claims 1 to 8, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement a human-computer interaction method for intelligent firefighting as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Virtual reality operation guidance and accompany training method and system based on virtual robot
CN109078334A
Immersive remote interaction system and method based on natural walking
CN111716365A
Method for optimizing human-computer interface layout of steel rail milling and grinding vehicle based on visual perception intensity
CN115220719A
Intelligent machine tool man-machine interaction interface layout optimization method based on cuckoo-particle swarm
CN116578296A
Voice call processing method and system, computer equipment and storage medium
CN118118597A
