A security check instruction analysis method and device based on human-computer interaction and a medium

Abstract

The application discloses a security check instruction analysis method and device based on human-computer interaction and a medium, relates to the technical field of human-computer interaction, and comprises the following steps: extracting an interactable object based on an interface evidence snapshot, forming an interactable object list, matching an instruction fragment set with the interactable object list, obtaining a target object and a single-step interaction script; binding the single-step interaction script and the target object, obtaining an effective execution token, driving the single-step interaction script to pre-execute the target object in an isolated environment that does not affect current business data according to the effective execution token, and obtaining a shadow operation result; and the application can solve the problem that a target object is easy to deviate from the actual interface context in the security check voice instruction analysis process by forming an interface evidence snapshot.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method, device and medium for parsing security inspection instructions based on human-computer interaction. Background Technology

[0002] With the informatization upgrade of security checks at airports, train stations, and ports, security operations are gradually evolving from manual click operations to voice-driven human-computer interaction. Conventional technologies typically employ fixed wake-up, speech recognition, semantic segmentation, interface control capture, and automated execution to assist in processing business operations such as querying, opening, writing, submitting, and reviewing, thereby improving operational efficiency and reducing the burden on manual operations.

[0003] Existing methods still have shortcomings. First, they lack joint constraints on the interface context, which can easily lead to deviations between the instruction parsing results and the actual target object. Second, they lack pre-execution verification, in-execution validation, and post-execution traceability mechanisms, making it difficult to balance operational security and audit traceability. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a security inspection instruction parsing method based on human-computer interaction to solve the problem that the target object is prone to deviating from the actual interface context.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a security check instruction parsing method based on human-computer interaction, comprising: Continuously acquire voice signals, and when the voice signal contains a preset wake word, trigger the screen state lock command, capture the complete rendering state of the current interface, generate an interface evidence snapshot, and acquire the command voice after the preset wake word. Then, convert and segment the command voice to obtain a set of command segments. Interactive objects are extracted based on the interface evidence snapshot, forming a list of interactive objects. The set of instruction fragments is matched with the list of interactive objects to obtain the target object and the single-step interaction script. The single-step interaction script and the target object are bound together to obtain a valid execution token. Based on the valid execution token, the single-step interaction script is driven to pre-execute the target object in an isolated environment that does not affect the current business data, so as to obtain the shadow operation result. The shadow operation result is compared with the target object and the current page identifier in the valid execution token to obtain the verification result. When the verification result is consistent, the formal operation is performed on the target object to obtain the formal execution result. Collect operator images, and associate operator images, target objects, valid execution tokens, shadow operation results, and formal execution results to form security check interaction audit records.

[0007] As a preferred embodiment of the security inspection instruction parsing method based on human-computer interaction described in this invention, the interface evidence snapshot includes: interface identifier, business stage identifier, focus control path, text of currently highlighted content, identifier of selected object, element attribute array of all interactive controls, platform region identifier, relative positional relationship of each control to the focus control, hierarchical relationship of interface controls, and complete path identifier of the focus control. The set of instruction fragments includes action fragments, object fragments, constraint fragments, and position fragments.

[0008] As a preferred embodiment of the security inspection instruction parsing method based on human-computer interaction described in this invention, the specific steps for extracting interactive objects based on interface evidence snapshots and forming an interactive object list are as follows: Read the control type and control state from the feature attribute array of all interactive controls in the interface evidence snapshot. When the control type belongs to the preset control type and the control state is available, use the feature attribute array of all corresponding interactive controls as interactive objects. Read the button title, field label, and record name corresponding to the interactive object from the feature attribute array of all interactive controls, and use them as the text information of the interactive object; Read the business type and bound business entity corresponding to the interactive object from the element attribute array of all interactive controls, and use them as the business information of the interactive object; Read the interface area, interface level, and relative position of each control to the focus control from the element attribute array of all interactive controls, and use them as the position anchoring information of the interactive object; The interactive objects, text information, business information, and location anchoring information are combined into a list of interactive objects.

[0009] As a preferred embodiment of the security inspection instruction parsing method based on human-computer interaction described in this invention, the specific steps for matching the instruction fragment set with the list of interactive objects are as follows: The object fragment is compared with the business information. When the object fragment is the same as the bound business entity, the corresponding interactive object is taken as the first candidate object. The location fragment is compared with the location anchor information. When the location fragment belongs to the location anchor information, the corresponding first candidate object is used as the second candidate object. Among all the second candidate objects, the second candidate object that is associated with the defined fragment is identified as the target object; Combine action fragments, constraint fragments, location fragments, and target objects into a single-step interactive script.

[0010] As a preferred embodiment of the security check instruction parsing method based on human-computer interaction described in this invention, the specific steps for obtaining a valid execution token are as follows: The target object is highlighted and displayed on the screen of the human-computer interaction platform, along with a confirmation prompt. Read the current page identifier, the current business stage identifier, and the current focused control path as the real-time business status before confirmation; The real-time business status before confirmation is compared with the corresponding information in the interface evidence snapshot to obtain the comparison result; When the comparison results match, receive a confirmation voice message from the security inspector; when the comparison results do not match, do not receive a confirmation voice message. After receiving confirmation from the security inspector, the target object, current page identifier, current focus control path, action fragment, and current time are combined into a valid execution token.

[0011] As a preferred embodiment of the security inspection instruction parsing method based on human-computer interaction described in this invention, the shadow operation result is obtained by using the current page identifier in the valid execution token as the pre-operation page identifier, the current focus control path as the pre-operation control path, the target object as the pre-operation target, and the action fragment as the pre-operation method to drive the page, and recording the expected hit object, the expected jump page, the expected changed field, and the expected field change result.

[0012] As a preferred embodiment of the security inspection instruction parsing method based on human-computer interaction described in this invention, the consistency verification of the shadow operation result is performed in the following specific steps: When the action fragment is any of the open, query, or replay, the expected hit object and the expected redirect page are compared with the target object and the current page identifier in the valid execution token to obtain the redirect consistency result. When the action fragment is any of the write, commit, or revert actions, the expected hit object and expected change fields are compared with the current business stage and target object to obtain the change consistency result. The jump consistency result and the change consistency result are used as the verification result.

[0013] As a preferred embodiment of the security inspection instruction parsing method based on human-computer interaction described in this invention, the operator image is obtained by capturing the current operator using the screen camera of the human-computer interaction platform when the formal execution result is obtained.

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the security inspection instruction parsing method based on human-computer interaction as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the security inspection instruction parsing method based on human-computer interaction as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By forming an interface evidence snapshot, the identification of the target object is established under the joint constraints of page state, focus control, business entity, and location anchoring information, which improves the accuracy of security check voice command parsing and the consistency of object positioning, and can solve the problem that the target object is prone to deviating from the actual interface context during the security check voice command parsing process; In addition, by constructing an execution closed loop through effective execution tokens, shadow operation results, consistency verification, and formal execution results, the security controllability of the operation process, the verifiability of the results, and the ability to trace and audit after the event are improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a security check instruction parsing method based on human-computer interaction.

[0019] Figure 2 A schematic diagram for generating interface evidence snapshots and instruction fragment sets.

[0020] Figure 3 This is a diagram illustrating the generation of the target object and the single-step interactive script.

[0021] Figure 4 This is a schematic diagram for verifying the consistency of shadow operation results. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a security check instruction parsing method based on human-computer interaction, including the following steps: S1: Continuously acquire voice signals. When the voice signal contains a preset wake-up word, trigger the screen state lock command, capture the complete rendering state of the current interface, generate an interface evidence snapshot, and acquire the command voice after the preset wake-up word. Convert and segment the command voice to obtain a set of command segments.

[0026] S1.1: The security inspector stands in front of the human-computer interaction platform and says a fixed wake-up word. After recognizing the fixed wake-up word, the human-computer interaction platform determines the moment of recognition of the fixed wake-up word as the lock moment and uses the display screen corresponding to the lock moment as the screen state lock object.

[0027] A fixed wake word is a voice word used to trigger the human-computer interaction platform to receive voice commands.

[0028] Read the interface identifier, business stage identifier, focus control path, text of currently highlighted content, identifier of selected object, element attribute array of all interactive controls, platform region identifier, and interface element layout information from the screen state lock object. Read the element attribute array of all interactive controls, the relative position relationship of each control to the focus control, the hierarchical relationship of interface controls, and the complete path identifier of the focus control from the interface element layout information.

[0029] The interface identifier, business stage identifier, focus control path, text of currently highlighted content, identifier of selected object, element attribute array of all interactive controls, platform region identifier, relative position of each control to the focus control, hierarchical relationship of interface controls, and complete path identifier of the focus control are taken as interface evidence snapshots.

[0030] It should also be noted that when the page is fully loaded, the human-computer interaction platform performs a traversal of the control tree identified by the current page from the root node to the leaf node, records each control node, and reads the control identifier, control type, control state, displayed text, parent control identifier, page area identifier, hierarchy number, relative position with the focused control, bound field identifier, and bound business record identifier.

[0031] Using the page identifier as the root path identifier, the parent control identifier, the current control identifier, and the hierarchy number are concatenated to form the complete path identifier of the focused control. The control identifier, control type, control state, display text, bound field identifier, bound business record identifier, page area identifier, hierarchy number, and complete path identifier of the focused control of all control nodes are merged into interface elements.

[0032] When the human-computer interaction platform detects page switching, focus switching, control state changes, or business record switching, update the interface elements.

[0033] S1.2: After the human-computer interaction platform generates a snapshot of the interface evidence, it receives the command voice, performs pre-emphasis processing on the command voice, and obtains pre-emphasized voice data, the expression of which is as follows: ; in, The first in the command voice The amplitude after pre-emphasis at each sampling point The first in the command voice The amplitude of each sampling point This is the pre-emphasis coefficient. The first in the command voice The amplitude of each sampling point This is the index of the sampling points in the command speech.

[0034] It should also be noted that in this embodiment, the pre-emphasis coefficient is 0.97, because 0.97 can maintain a relatively stable balance between the high-frequency enhancement effect and the noise amplification degree. A value greater than 0.97 is likely to cause speech feature distortion and reduce the stability of subsequent recognition, while a value less than 0.97 is likely to cause the enhancement of high-frequency information such as consonants to be insignificant, thereby reducing the distinguishability of speech features.

[0035] The pre-emphasized speech data is segmented sequentially to obtain a speech frame sequence. The speech frame sequence is then windowed to obtain a windowed speech frame sequence, as shown in the following expression: ; in, For the first The amplitude after windowing at each sampling point For the speech frame sequence number The amplitude of each sampling point The total number of sampling points in each frame. This is the index of each frame's sampling point.

[0036] All windowed amplitude values ​​of each frame are merged into a windowed speech frame in chronological order, and all windowed speech frames are merged into a windowed speech frame sequence in chronological order.

[0037] It should also be noted that, and It is the standard constant of the Hamming window, used to achieve a trade-off between the main lobe width and the side lobe suppression capability.

[0038] The short-time energy and zero-crossing rate are calculated frame-by-frame based on the speech frame sequence, as shown in the following expressions: ; ; in, For short-term energy, For zero-crossing rate, It is a sign function, taking -1 when the magnitude is negative and 1 when the magnitude is non-negative.

[0039] Based on short-time energy, a high threshold and a low threshold for short-time energy are set. For example, 30% of the maximum value of short-time energy is set as the high threshold for short-time energy, and 10% of the maximum value of short-time energy is set as the low threshold for short-time energy.

[0040] It should also be noted that the short-term energy threshold is set at 30% because 30% can reliably distinguish the effective main speech segment from background noise and weak onset sounds. If it is greater than 30%, it is easy to miss weak onset sounds, weak tail sounds, and low-volume speech frames, causing the coarse start point of the speech segment to shift backward and the coarse end point to shift forward, thus shortening the effective speech body. If it is less than 30%, it is easy to misjudge background noise, breathing sounds, or environmental disturbances as effective main speech segments, resulting in an expansion of the coarse localization range of the speech segment and an increase in the probability of false detection.

[0041] It should also be noted that the short-term energy threshold is set at 10% because 10% can preserve low-energy transitional speech forward and backward on the basis of coarse localization, taking into account endpoint integrity and noise resistance stability. If it is greater than 10%, it is easy to lose low-energy consonants, neutral sounds and codas on both sides of the main speech segment, resulting in incomplete endpoint preservation. If it is less than 10%, when backtracking forward and expanding backward, more silent frames and low-amplitude noise frames will be included in the speech segment, resulting in a longer endpoint range and reduced purity of subsequent feature extraction.

[0042] Starting from the beginning of the speech frame sequence, a fixed number of frames (ten frames in this embodiment) are selected as a reference range. Speech frames with short-time energy below the short-time energy threshold are selected from the reference range. The zero-crossing rate threshold is set as twice the average zero-crossing rate of the selected speech frames.

[0043] S1.3: Determine the starting frame and ending frame of a speech segment based on the short-time energy high threshold, short-time energy low threshold and zero-crossing rate threshold: Take the first speech frame with short-time energy not lower than the short-time energy high threshold as the coarse starting frame, and backtrack frame by frame from the coarse starting frame to record the first speech frame with short-time energy less than the short-time energy low threshold, and take the corresponding next frame as the energy starting frame.

[0044] Starting from the energy starting frame, backtrack frame by frame, record the first zero-crossing rate that does not exceed the zero-crossing rate threshold, and take the corresponding next frame as the starting frame of the speech segment.

[0045] Scan the speech frame sequence from back to front, take the last speech frame whose short-time energy is not lower than the short-time energy high threshold as the coarse endpoint frame, start searching backward from the coarse endpoint frame, record the first speech frame whose short-time energy is lower than the short-time energy low threshold, and take the corresponding previous frame as the energy endpoint frame.

[0046] Starting from the energy endpoint frame, check backwards and record that the last zero-crossing rate does not exceed the zero-crossing rate threshold. Then, take the corresponding previous frame as the speech segment endpoint frame.

[0047] The windowed speech frames from the start frame to the end frame of the speech segment in the windowed speech frame sequence are merged into a valid speech frame sequence in chronological order.

[0048] S1.4: Perform Fast Fourier Transform on the valid speech frame sequence: Pad each valid speech frame with zeros to 512 sampling points, and perform Fast Fourier Transform on the zero-padding speech frames to obtain the power spectrum value sequence of each valid speech frame.

[0049] It should also be noted that zeroing is added to each effective speech frame to bring the total number of sampling points to 512 because 512 can balance computational efficiency and the stability of subsequent Mel filter processing. A value greater than 512 would increase computational overhead but would not improve the speech command recognition effect, while a value less than 512 would result in insufficient frequency resolution.

[0050] Perform Mel filtering on the power spectrum sequence to obtain the output energy, as shown in the following expression: ; in, In order to output energy, This is the index of the frequency points after the Fast Fourier Transform. This represents the total number of frequency points after the Fast Fourier Transform. For the first Power spectrum values ​​at each frequency point For the first Mel filter coefficients at each frequency point.

[0051] The natural logarithm of the output energy is taken to obtain the log-Mel filter bank energy. All log-Mel filter bank energies are then merged into a log-Mel filter bank energy sequence.

[0052] Perform a discrete cosine transform on the log-Mel filter bank energy sequence of each frame to obtain the Mel frequency cepstral coefficient sequence. Concatenate the Mel frequency cepstral coefficient sequence and the log-frame energy to form a static feature. Calculate the first-order difference and second-order difference of the static feature along the time axis.

[0053] Take the natural logarithm of the short-time energy of each frame to obtain the logarithmic frame energy.

[0054] Static features, first-order differences, and second-order differences are concatenated into a single-frame feature vector. All single-frame feature vectors are then concatenated in chronological order to obtain the speech feature vector.

[0055] S1.5: Training the Hidden Markov Model: The Hidden Markov Model includes an initial non-emitting state, a first emitting state, a second emitting state, a third emitting state, and a terminating non-emitting state. The initial non-emitting state can only transition to the first emitting state. The three emitting states can only remain in the current emitting state or transition to the next emitting state. The terminating non-emitting state can no longer transition outwards.

[0056] We collected action words, object words, qualifiers, and position words read aloud by humans as training samples.

[0057] Calculate the speech feature vector corresponding to each word in the training sample as the training feature vector according to steps S1.2-S1.4.

[0058] The training feature vector is divided into three segments to obtain the first training vector, the second training vector, and the third training vector.

[0059] The average value of the single-frame feature vectors contained in the first training vector is used as the mean vector of the first emission state.

[0060] The average value of the single-frame feature vectors contained in the second training vector is used as the mean vector of the second emission state.

[0061] The average value of the single-frame feature vectors contained in the third training vector is used as the mean vector of the third emission state.

[0062] The difference between the single-frame feature vector and the corresponding mean vector in the three training vectors is taken as the bias vector. The mean of the squares of all elements in the bias vector is taken as the single-frame variance. All single-frame variances of each transmission state are arranged diagonally to obtain the corresponding diagonal covariance matrix.

[0063] The mean vector and diagonal covariance matrix corresponding to the three emission states are used as the initial values ​​of the observation probability distribution parameter set.

[0064] The single-frame feature vectors in the training feature vector are arranged in chronological order to form an observation sequence. The observation sequence is then input into a Hidden Markov Model, and the forward and backward probabilities of the observation sequence are calculated as follows: ; ; in, For the first The time is located at the Forward probability of each emission state For the first Time of the first The forward probability of each state For from the first The state to the 1st The state transition probability of each emission state. For the first The launch state for the first The observation probability of a single-frame feature vector at time step [time]. For the first The single-frame feature vector at time step, For state index, For the launch status index, For time index, For the first The time is located at the The backward probability of each state. For the first The launch state for the first The observation probability of a single-frame feature vector at time step [time]. For the first The single-frame feature vector at time step, For the first The time is located at the The backward probability of each emission state.

[0065] The state occupancy probability of each transmission state at each time step is calculated using the forward and backward probabilities, as shown in the following expression: ; in, For the first The time is located at the The state occupancy probability of each emission state.

[0066] The transition probability between adjacent states is calculated using the following expression: ; in, For the first The time is located at the The state transitions to the first state. The transition probability of each emission state.

[0067] The state transition probability is updated using the following expression: ; in, For the updated version from the first The state to the 1st The state transition probability of each emission state. For the first The training sample of the th training sample The time is located at the The state transitions to the first state. The transition probability of each emission state. For the first The training sample of the th training sample The time is located at the The state occupancy probability of each state.

[0068] The mean vector for each emission state is updated as follows: ; in, For the first The mean vector of each updated launch state For the first The training sample of the th training sample The single-frame feature vector at time step, For the first The training sample of the th training sample The time is located at the The state occupancy probability of each state.

[0069] The diagonal covariance matrix for each emission state is updated as follows: ; in, For the first The diagonal covariance matrix after the launch state update For the first The mean vector of each emission state. This indicates transpose.

[0070] The corresponding parameters of the Hidden Markov Model are replaced with the updated state transition probabilities, mean vector, and diagonal covariance matrix to obtain the trained Hidden Markov Model.

[0071] Each word in the training samples corresponds to a trained Hidden Markov Model.

[0072] S1.6: Connect the head-to-tail Hidden Markov Models corresponding to action words, object words, qualifiers and position words in the trained word-level Hidden Markov Model to form candidate instruction paths.

[0073] For each candidate instruction path, Viterbi decoding is performed, and the maximum cumulative probability of the speech feature vector being transferred frame by frame along the candidate instruction path and the observation being transmitted is calculated, as shown in the following expression: ; in, For the first Time reaches the The maximum cumulative probability of each launch state This indicates taking the maximum value. For the first Time reaches the The maximum cumulative probability of each launch state For the updated version from the first The state to the 1st The state transition probability of each emission state. For the first The launch state for the first The observation probability of a single-frame feature vector at time step [time]. For the first The single-frame feature vector at time step, and This is the index of the launch state in the candidate command path. The frame index for the speech feature vector.

[0074] The maximum cumulative termination probability of a candidate instruction path is calculated using the following expression: ; in, This represents the maximum cumulative probability of termination for a candidate instruction path. For the first Frame 1 The maximum cumulative probability of each launch state For the updated version from the first The state to the 1st The transition probability of terminating a non-emission state. This represents the last frame of the speech feature vector. This is the last non-launch state of the candidate instruction path.

[0075] The candidate instruction path corresponding to the maximum cumulative termination probability is taken as the optimal instruction path, and the action words, object words, qualifier words and position words corresponding to the Hidden Markov Model in the optimal instruction path are concatenated into the instruction text.

[0076] S1.7: Treat action words in the instruction text as action fragments, object words as object fragments, qualifiers as qualifier fragments, and position words as position fragments.

[0077] Combine action fragments, object fragments, bound fragments, and position fragments into a set of instruction fragments.

[0078] It should also be noted that by locking the screen state of the human-computer interaction platform to form an interface evidence snapshot, and converting and segmenting the command voice to obtain a set of command fragments, the determination of the subsequent target object is based on the interface state represented by the interface identifier, business stage identifier, focus control path, and the text of the currently highlighted content. This reduces the possibility of the command fragment set being matched out of the current interface state during the security inspection command parsing process.

[0079] S2: Extract interactive objects based on the interface evidence snapshot, form a list of interactive objects, match the set of instruction fragments with the list of interactive objects, and obtain the target object and the single-step interaction script.

[0080] S2.1: Read the control type and control state item by item from the feature attribute array of all interactive controls in the interface evidence snapshot in the order of display; when the control type belongs to the preset control type and the control state is available, determine the feature attribute array of all corresponding interactive controls as interactive objects.

[0081] Preset control types refer to the control types used to trigger "open, query, replay, submit, withdraw, switch pages, retrieve records, select objects, and fill in conclusions".

[0082] In this embodiment, the preset control types include button controls, tab controls, menu controls, list item controls, record item controls, input box controls, drop-down selection controls, checkboxes, radio buttons, image retrieval controls, video playback controls, submit controls, undo controls, query controls, and page navigation controls.

[0083] S2.2: According to the display order of interactive objects in the feature attribute array of all interactive controls, read the button title, field label and record name corresponding to each interactive object one by one from the feature attribute array of all interactive controls, and use the button title, field label and record name as the text information of the corresponding interactive object.

[0084] Read the business type and bound business entity corresponding to each interactive object one by one from the element attribute array of all interactive controls, and use the business type and bound business entity as the business information of the corresponding interactive object.

[0085] Read the interface area, interface level, and relative position of each control to the focus control from the element attribute array of all interactive controls one by one, and use the interface area, interface level, and relative position of each control to the focus control as the position anchoring information of the corresponding interactive object.

[0086] The interactive objects, text information, business information, and location anchoring information are combined into a list of interactive objects.

[0087] S2.3: Read object fragments from the instruction fragment set and compare each object fragment with the business information in the list of interactive objects. When an object fragment is the same as the bound business entity, the corresponding interactive object is determined as the first candidate object.

[0088] The bound business entity refers to the type of business object that the interactive object corresponds to in the security inspection business.

[0089] The location fragments are read from the instruction fragment set, and the location fragments are compared with the location anchoring information corresponding to the first candidate object item by item. When the location fragment can find the corresponding location attribute in the location anchoring information, and the location attribute is the same as the location range, hierarchical position or relative orientation represented by the location fragment, the corresponding first candidate object is determined as the second candidate object.

[0090] Read the specified fragment from the instruction fragment set, and compare the specified fragment with the text of the currently highlighted content in the interface evidence snapshot item by item. When the business content corresponding to the specified fragment and the text of the currently highlighted content in the interface evidence snapshot are the same, the corresponding second candidate object is determined as the target object.

[0091] Combine action fragments, constraint fragments, location fragments, and target objects into a single-step interactive script.

[0092] It should also be noted that before filtering target objects, object fragments, location fragments, and limiting fragments need to be standardized: read the object word mapping table, location word mapping table, and limiting word mapping table from the human-computer interaction platform, and then standardize and replace the object fragments, location fragments, and limiting fragments.

[0093] The object term mapping table stores the standard name, synonyms, abbreviations, colloquial terms and easily confused terms corresponding to the bound business entity. The location term mapping table stores the standard location labels corresponding to the interface area, hierarchical position and relative orientation. The qualifier term mapping table stores the standard qualifier labels corresponding to the business content in the highlighted record.

[0094] It should also be noted that by extracting interactive objects based on interface evidence snapshots and forming a list of interactive objects, the set of instruction fragments is matched with the list of interactive objects to obtain the target object and the single-step interaction script. This ensures that the determination of the target object is simultaneously constrained by business information, location anchoring information, and the text of the currently highlighted content, reducing the possibility of operational errors being applied to non-target objects during the security check instruction parsing process.

[0095] S3: Bind the single-step interaction script and the target object to obtain a valid execution token. Based on the valid execution token, drive the single-step interaction script to pre-execute the target object in an isolated environment that does not affect the current business data, and obtain the shadow operation result.

[0096] S3.1: Locate the display position of the target object in the current display screen of the human-computer interaction platform, and highlight the target object on the screen of the human-computer interaction platform.

[0097] A confirmation prompt is displayed at the top of the screen of the human-computer interaction platform. The confirmation prompt is used to remind the security inspector that the corresponding action will be performed on the target object according to the single-step interaction script.

[0098] After displaying the confirmation prompt, read the current page identifier, current business stage, current focus control path, and current highlighted record as the real-time status before confirmation.

[0099] S3.2: Compare the current page with the interface identifier in the interface evidence snapshot. If the current page identifier is the same as the interface identifier, the page comparison result is determined to be consistent. If the current page identifier is different from the interface identifier, the page comparison result is determined to be inconsistent.

[0100] The current business stage is compared with the business stage identifier in the interface evidence snapshot. If the current business stage and the business stage identifier are the same, the business stage comparison result is determined to be consistent. If the current business stage and the business stage identifier are different, the business stage comparison result is determined to be inconsistent.

[0101] The current focus control path is compared with the focus control path in the interface evidence snapshot. If the current focus control path is the same as the focus control path, the focus control comparison result is determined to be consistent. If the current focus control path is different from the focus control path, the focus control comparison result is determined to be inconsistent.

[0102] The text of the currently highlighted record is compared with the text of the currently highlighted content in the interface evidence snapshot. If the text of the current highlighted record is the same as the text of the current highlighted content, the comparison result of the highlighted record is determined to be consistent. If the text of the current highlighted record is different from the text of the current highlighted content, the comparison result of the highlighted record is determined to be inconsistent.

[0103] The comparison result is considered consistent when the page comparison result, business stage comparison result, focus control comparison result, and highlighted record comparison result are all consistent. The comparison result is considered inconsistent when any one of these four results is inconsistent.

[0104] S3.3: When the comparison results are consistent, the human-computer interaction platform receives the security inspector's confirmation voice while the confirmation prompt is displayed, merges the target object, the current page identifier, the current focus control path, the action fragment, and the current time to obtain a valid execution token. When the comparison results are inconsistent, the subsequent binding process corresponding to the current confirmation prompt is stopped.

[0105] S3.4: Perform pre-operations according to the action fragments, limit fragments, position fragments, and target objects recorded in the single-step interaction script: use the current page identifier in the valid execution token as the pre-operation page identifier, the current focus control path as the pre-operation control path, the target object as the pre-operation target, and the action fragment as the pre-operation method to drive the page. When the action fragment indicates opening, querying, or replaying, record the expected hit object and expected jump page after page driving as the shadow operation result. When the action fragment is writing, committing, or retracting, record the expected hit object, expected changed field, and expected field change result after page driving as the shadow operation result.

[0106] It should also be noted that after obtaining a valid execution token, a pre-operation copy is created using the current page identifier, the current focused control path, and the target object in the valid execution token. The pre-operation copy uses the same page structure, control properties, page navigation rules, and field validation rules as the current page, but the write, commit, and retraction results in the pre-operation copy are only written to the memory cache and are not committed to the business database.

[0107] Shadow operation refers to a pre-execution process in which a single-step interactive script is executed in a pre-operation copy environment built on a human-computer interaction platform based on a valid execution token. This process involves page-driven and rule-based deductions that are the same as those in the actual operation, but it prohibits submission to the business database, triggering process flow, calling external interfaces, and sending message notifications.

[0108] It should also be noted that by binding the single-step interactive script and the target object to obtain a valid execution token, and performing shadow operations on the single-step interactive script to obtain shadow operation results, the confirmation of the current page identifier, the current business stage, the current focus control path, and the current highlighted record, as well as the pre-verification of the operation results, are completed before the formal operation, reducing the possibility of continuing to execute subsequent operations when the interface state changes.

[0109] S4: Compare the shadow operation result with the target object and the current page identifier in the valid execution token to obtain the verification result. When the verification result is consistent, perform the formal operation on the target object to obtain the formal execution result.

[0110] S4.1: When the action fragment is any of the open, query, or replay, compare the expected hit object with the target object in the valid execution token. If the expected hit object and the target object are the same, the object comparison result is determined to be consistent. If the expected hit object and the target object are different, the object comparison result is determined to be inconsistent.

[0111] The system reads the target page navigation relationship corresponding to the current page identifier from the human-computer interaction platform, compares the expected navigation page with the target page navigation relationship corresponding to the current page identifier, and determines that the page comparison result is consistent when the expected navigation page belongs to the target page navigation relationship corresponding to the current page identifier, and determines that the page comparison result is inconsistent when the expected navigation page does not belong to the target page navigation relationship corresponding to the current page identifier.

[0112] When both the object comparison result and the page comparison result are consistent, the redirection consistency result is determined to be consistent. When either the object comparison result or the page comparison result is inconsistent, the redirection consistency result is determined to be inconsistent.

[0113] When the action fragment is any of write, commit, or retract, the expected hit object is compared with the target object in the valid execution token. If the expected hit object is the same as the target object, the object comparison result is determined to be consistent. If the expected hit object is different from the target object, the object comparison result is determined to be inconsistent.

[0114] The system reads the field change permission relationship corresponding to the current business stage from the human-computer interaction platform, compares the expected change field with the field change permission relationship corresponding to the current business stage, and determines that the field comparison result is consistent when the expected change field belongs to the field change permission relationship corresponding to the current business stage, and determines that the field comparison result is inconsistent when the expected change field does not belong to the field change permission relationship corresponding to the current business stage.

[0115] When both the object comparison result and the field comparison result are consistent, the change consistency result is determined to be consistent. When either the object comparison result or the field comparison result is inconsistent, the change consistency result is determined to be inconsistent.

[0116] It should also be noted that the target page jump relationship is obtained by collecting the source page identifier, trigger control identifier, action type and target page identifier as a page jump relationship table, using the current page identifier in the valid execution token as the source page identifier, and the action fragment as the action type, and retrieving the corresponding target page identifier from the page jump relationship table.

[0117] The field change permission relationship is obtained by collecting business stage identifiers, action types, permitted change field identifiers, and field validation rules. The current business stage in the valid execution token is used as the business stage identifier, and the action fragment is used as the action type. The permitted change field identifiers are retrieved from the field change permission relationship table.

[0118] Consistency verification refers to verifying the consistency between the shadow operation result and the valid execution token.

[0119] S4.2: When the action fragment is open, query, or replay, the jump consistency result is used as the verification result; when the action fragment is write, commit, or retract, the change consistency result is used as the verification result.

[0120] If the verification result is inconsistent, stop performing formal operations on the target object.

[0121] When the verification result is consistent and the action fragment is open, the page opening operation is performed according to the current page identifier, current focus control path and target object in the valid execution token, and the display result after the page is opened is recorded as the official execution result; When the verification result is consistent and the action fragment is a query, the query operation is performed according to the current page identifier, the current focused control path and the target object in the valid execution token, and the query return content is recorded as the formal execution result; When the verification result is consistent and the action fragment is a playback, the playback operation is performed according to the current page identifier, current focus control path and target object in the valid execution token, and the playback return content is recorded as the formal execution result.

[0122] When the verification result is consistent and the action fragment is written, perform the field writing operation according to the current business stage, current focus control path and target object in the valid execution token, and record the content after the field is written as the formal execution result; When the verification result is consistent and the action fragment is submitted, the submission operation is performed according to the current business stage, current focus control path and target object in the valid execution token, and the business status content after submission is recorded as the formal execution result; When the verification result is consistent and the action fragment is withdrawn, the withdrawal operation is performed according to the current business stage, current focus control path and target object in the valid execution token, and the business status content after withdrawal is recorded as the formal execution result.

[0123] It should also be noted that by performing consistency verification on the shadow operation results based on the valid execution token, and then performing formal operations on the target object according to the verification results to obtain the formal execution results, the open, query, and replay operations are constrained by the target page jump relationship, and the write, submit, and retract operations are constrained by the field change permission relationship corresponding to the current business stage. This improves the accuracy and security controllability of the formal operations after the security inspection command is parsed.

[0124] S5: Collect operator images, associate operator images, target objects, valid execution tokens, shadow operation results, and formal execution results to form security check interaction audit records.

[0125] The moment the formal execution result is formed is used as the snapshot moment. The screen camera of the human-computer interaction platform is used to capture the current operator and obtain the operator's image.

[0126] The operator image, target object, valid execution token, shadow operation result, and formal execution result are merged into a security check interaction audit record.

[0127] It should also be noted that by collecting operator images and associating operator images, target objects, valid execution tokens, shadow operation results, and formal execution results to form security check interaction audit records, a corresponding relationship is established between security check instruction parsing, shadow operation results, consistency verification, and formal operations, thereby improving the verifiability of the security check operation process and the ability to trace back after the fact.

[0128] The snapshots and data collection performed in this embodiment were all done with the consent of the parties involved and were used for legitimate purposes.

[0129] This embodiment also provides a computer device applicable to the security inspection instruction parsing method based on human-computer interaction, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the security inspection instruction parsing method based on human-computer interaction as proposed in the above embodiment.

[0130] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0131] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the security inspection instruction parsing method based on human-computer interaction as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0132] In summary, this invention improves the accuracy of security check voice command parsing and the consistency of object positioning by creating an interface evidence snapshot, enabling target object identification to be based on the joint constraints of page state, focus control, business entity, and location anchoring information. This solves the problem that target objects are prone to deviating from the actual interface context during security check voice command parsing. Furthermore, by constructing an execution closed loop through effective execution tokens, shadow operation results, consistency verification, and formal execution results, it enhances the security and controllability of the operation process, the verifiability of the results, and the ability to conduct post-event audit traceability.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A security check instruction parsing method based on human-computer interaction, characterized in that: include, Continuously acquire voice signals, and when the voice signal contains a preset wake word, trigger the screen state lock command, capture the complete rendering state of the current interface, generate an interface evidence snapshot, and acquire the command voice after the preset wake word. Then, convert and segment the command voice to obtain a set of command segments. Interactive objects are extracted based on the interface evidence snapshot, forming a list of interactive objects. The set of instruction fragments is matched with the list of interactive objects to obtain the target object and the single-step interaction script. The single-step interaction script and the target object are bound together to obtain a valid execution token. Based on the valid execution token, the single-step interaction script is driven to pre-execute the target object in an isolated environment that does not affect the current business data, so as to obtain the shadow operation result. The shadow operation result is compared with the target object and the current page identifier in the valid execution token to obtain the verification result. When the verification result is consistent, the formal operation is performed on the target object to obtain the formal execution result. Collect operator images, and associate operator images, target objects, valid execution tokens, shadow operation results, and formal execution results to form security check interaction audit records.

2. The security check instruction parsing method based on human-computer interaction as described in claim 1, characterized in that, The interface evidence snapshot includes: interface identifier, business stage identifier, focus control path, text of currently highlighted content, identifier of selected object, element attribute array of all interactive controls, platform region identifier, relative positional relationship of each control to the focus control, hierarchical relationship of interface controls, and complete path identifier of the focus control. The set of instruction fragments includes action fragments, object fragments, constraint fragments, and position fragments.

3. The security inspection command parsing method based on human-computer interaction as described in claim 1, characterized in that, The specific steps for extracting interactive objects based on interface evidence snapshots and forming a list of interactive objects are as follows: Read the control type and control state from the feature attribute array of all interactive controls in the interface evidence snapshot. When the control type belongs to the preset control type and the control state is available, take the feature attribute array of all corresponding interactive controls as the interactive object. Read the button title, field label, and record name corresponding to the interactive object from the feature attribute array of all interactive controls, and use them as the text information of the interactive object; Read the business type and bound business entity corresponding to the interactive object from the element attribute array of all interactive controls, and use them as the business information of the interactive object; Read the interface area, interface level, and relative position of each control to the focus control from the element attribute array of all interactive controls, and use them as the position anchoring information of the interactive object; The interactive objects, text information, business information, and location anchoring information are combined into a list of interactive objects.

4. The security inspection command parsing method based on human-computer interaction as described in claim 1, characterized in that, The specific steps for matching the set of instruction fragments with the list of interactive objects are as follows: The object fragment is compared with the business information. When the object fragment is the same as the bound business entity, the corresponding interactive object is taken as the first candidate object. The location fragment is compared with the location anchor information. When the location fragment belongs to the location anchor information, the corresponding first candidate object is used as the second candidate object. Among all the second candidate objects, the second candidate object that is associated with the defined fragment is identified as the target object; Combine action fragments, constraint fragments, location fragments, and target objects into a single-step interactive script.

5. The security inspection command parsing method based on human-computer interaction as described in claim 1, characterized in that, The specific steps to obtain a valid execution token are as follows: The target object is highlighted and displayed on the screen of the human-computer interaction platform, along with a confirmation prompt. Read the current page identifier, the current business stage identifier, and the current focused control path as the real-time business status before confirmation; The real-time business status before confirmation is compared with the corresponding information in the interface evidence snapshot to obtain the comparison result; When the comparison results match, receive a confirmation voice message from the security inspector; when the comparison results do not match, do not receive a confirmation voice message. After receiving confirmation from the security inspector, the target object, current page identifier, current focus control path, action fragment, and current time are combined into a valid execution token.

6. The security inspection command parsing method based on human-computer interaction as described in claim 1, characterized in that, The shadow operation result is obtained by using the current page identifier in the valid execution token as the pre-operation page identifier, the current focus control path as the pre-operation control path, the target object as the pre-operation target, and the action fragment as the pre-operation method to drive the page, and recording the expected hit object, the expected jump page, the expected changed field, and the expected field change result.

7. The security inspection command parsing method based on human-computer interaction as described in claim 1, characterized in that, The consistency verification of the shadow operation results is performed as follows: When the action fragment is any of the open, query, or replay, the expected hit object and the expected redirect page are compared with the target object and the current page identifier in the valid execution token to obtain the redirect consistency result. When the action fragment is any of the write, commit, or revert actions, the expected hit object and expected change fields are compared with the current business stage and target object to obtain the change consistency result. The jump consistency result and the change consistency result are used as the verification result.

8. The security check instruction parsing method based on human-computer interaction as described in claim 1, characterized in that, The operator image is captured by the screen camera of the human-computer interaction platform when the formal execution result is obtained.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the security inspection instruction parsing method based on human-computer interaction as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the security inspection instruction parsing method based on human-computer interaction as described in any one of claims 1 to 8.

Images