Multi-modal collaborative instruction analysis and execution method and system for sound box with screen

Abstract

The application discloses a multi-modal cooperative instruction analysis and execution method and system for a screen-equipped sound box, relates to the technical field of intelligent sound box interaction control, and comprises the following steps: merging interactive response inputs generated by a local end of the screen-equipped sound box into interactive acceptance events according to the entering order of the local interactive processing procedure, generating acceptance time sequence positions for the interactive acceptance events, and forming an acceptance time sequence chain; forming a display bearing relationship according to the corresponding relationship between display objects and action entrances, associating the display bearing relationship with the corresponding acceptance time sequence positions, writing the acceptance time sequence positions into the acceptance time sequence chain, and forming a display bearing state record. According to the application, the interactive acceptance events and the acceptance time sequence chain are established at the local end, so that the screen trigger input, the voice input and the display bearing state can be associated and processed on the basis of the same acceptance time sequence, and the instruction acceptance accuracy of the screen-equipped sound box in continuous multi-round interaction is improved.

Description

Technical Field

[0001] This invention relates to the field of smart speaker interactive control technology, specifically to a multimodal collaborative instruction parsing and execution method and system for speakers with screens. Background Technology

[0002] Screen-based smart speakers typically combine screen display, touch input, voice input, and local action execution capabilities. In actual interaction, users can first trigger objects on the display interface via the screen, and then express content through short voice input actions, allowing voice input and screen display content to together form a complete operation intention. For example, when there are multiple operable objects on the display interface, short voice messages often only express actions such as "open," "play," "confirm," and "cancel," while the operable object needs to be confirmed in conjunction with the objects already triggered or currently displayed on the screen. In multimodal interaction scenarios, screen-triggered input, voice input, changes in displayed content, and calls to local action entry points typically form stage data through different processing entry points. Because there are sequential relationships between different inputs entering the local processing flow, the display refresh state, and the voice parsing results, determining the execution object solely based on a single voice input or the current display state may not accurately reflect the sequential relationship between screen-triggered input and voice input. For speakers with screens, a unified sequential basis for different interactive inputs needs to be established locally to determine whether there is a sequential relationship between the displayed object, the screen-triggered object, and the short voice action. Furthermore, within the same display interface or a continuous display interface, the displayed object may be maintained, refreshed, replaced, or the action entry point may change. Whether the same object can still be used as a short voice completion object under different succession sequences cannot be determined solely by the display object name or the current interface position. It is also necessary to confirm this by combining the display carrying status, the object carrying continuity, and the correspondence between the short voice action and the action entry point. At the same time, screen-triggered input and voice input may originate from different entities. If the consistency of their entity succession is not determined, it may also lead to the short voice input being incorrectly succeeded to the screen-triggered object of other entities. Therefore, this invention proposes a multimodal collaborative instruction parsing and execution method and system for speakers with screens. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal collaborative instruction parsing and execution method and system for speakers with screens, so as to solve the problems mentioned in the background art.

[0004] This invention can be achieved through the following technical solution: a multimodal cooperative instruction parsing and execution method for speakers with screens, comprising: Step 1: Group the interactive response inputs generated by the local terminal of the speaker with screen into interactive acceptance events according to the order in which they enter the local interactive processing flow, generate acceptance sequence positions for the interactive acceptance events, and form an acceptance sequence chain. Step 2: Based on the correspondence between the displayed object and the action entry point, form a display carrying relationship, associate the display carrying relationship with the corresponding receiving sequence position and write it into the receiving sequence chain, and form a display carrying status record; based on the target receiving sequence position, determine the valid display carrying status record in the display carrying status record; Step 3: Compare the interactive acceptance events with screen trigger attributes with the valid display acceptance status records corresponding to their acceptance sequence positions to determine the screen trigger object and form a screen trigger acceptance relationship that includes the screen trigger object, the acceptance sequence position, and the screen trigger subject identifier. Step 4: Analyze the acceptance requirements of interactive acceptance events with voice input attributes; when the content expressed by the object needs to be completed by the display object and the content expressed by the action belongs to the preset short voice acceptance action, the acceptance sequence position of the interactive acceptance event is used as the voice acceptance sequence position, and a short voice acceptance relationship containing the voice acceptance sequence position and the voice subject identifier is formed. Step 5: Call the valid display carrying status record according to the voice receiving sequence position; based on the valid display carrying status record, screen trigger receiving relationship and short voice receiving relationship, determine the display receiving consistency and the main receiving consistency, and form the instruction execution result or receiving execution blocking result according to the determination result.

[0005] A further technical improvement of the present invention is that the method for determining the basis for generating the succession sequence position in step one includes: Read the stage data formed by the interactive response input through different processing entry points in the local interactive processing flow, and obtain the processing entry identifier, trigger source identifier and stage formation position corresponding to each stage data; Read the entry acceptance sequence record, which stores the adjacent acceptance relationship between different processing entry identifiers; Stage data with the same trigger source identifier are grouped into the same merge sequence, and the stage data in the same merge sequence are sorted according to the stage formation position. The first-ranked stage data in the same sequence to be merged is used as the starting stage data for an interactive event. Read two adjacent stages of data in the same sequence to be merged along the sorting direction, and compare the processing entry identifiers corresponding to the two stages with the entry acceptance order records; When the processing entry identifiers corresponding to two stages of data have an adjacent entry relationship, and the stage formation position corresponding to the later stage data is later than the stage formation position corresponding to the earlier stage data, the later stage data is written into the interaction acceptance event to which the earlier stage data belongs. When the processing entry identifiers corresponding to two stages of data do not have an adjacent succession relationship, or when the stage formation position corresponding to the later stage data is not later than the stage formation position corresponding to the earlier stage data, the later stage data will be used as the starting stage data of the new interactive succession event. The earliest stage data in the same interactive event is used as the basis for generating the timing sequence of the interactive event.

[0006] A further technical improvement of the present invention is that: the method for generating succession sequence positions based on succession sequence position generation criteria and writing them into the succession sequence chain in step one includes: Read multiple interactive acceptance events whose acceptance sequence positions have been determined as the basis for generation, and read the preset initial position value and preset acceptance interval value. The preset acceptance interval value is greater than zero. For each interactive acceptance event, the ordering of events is based solely on the order of the stages corresponding to the acceptance sequence of that interactive acceptance event. Multiple interactive events are arranged in order of priority based on the sequence of events; The order of the first interactive event is set as the preset initial position value. Starting from the first interactive event in the sequence, read the sequence position of the previous interactive event in sequence, and add the sequence position of the previous interactive event to the preset interval value to determine the sequence position of the current interactive event. Write the sequence order of each interactive event into the sequence chain, and write the sequential relationship between adjacent interactive events into the sequence chain.

[0007] A further technical improvement of the present invention is that the method for forming a display bearing status record in step two includes: When the current acceptance sequence position is not the first acceptance sequence position in the acceptance sequence chain, read the display carrying relationship corresponding to the current acceptance sequence position, as well as the previous display carrying status record that is adjacent to the current acceptance sequence position and whose acceptance sequence position is earlier than the current acceptance sequence position. Read each display object in the display carrying relationship in sequence, and take the currently read display object as the current display object, and read the action entry corresponding to the current display object; Compare the currently displayed object and its corresponding action entry with the displayed object and action entry in the previously displayed carrying state record; When there is a corresponding display object in the prior display status record that is identical to the current display object and has the same action entry point, the object carrier identifier of the corresponding display object in the prior display status record is used as the object carrier identifier of the current display object. When there is no corresponding display object in the prior display status record that matches the current display object, or when there is a corresponding display object that matches the current display object but the action entry is different, a new object carrier identifier is generated for the current display object. After determining the object carrier identifier for each display object in the display carrier relationship, the current acceptance sequence position, display carrier relationship, and object carrier identifier of each display object are associated and written to form a display carrier status record corresponding to the current acceptance sequence position.

[0008] A further technical improvement of the present invention is that the method for determining the valid display bearing status record in step two includes: Read the target acceptance sequence position, and read the display acceptance status record whose corresponding acceptance sequence position is not later than the target acceptance sequence position; Read the display bearer status records one by one, and use the currently read display bearer status record as the current display bearer status record; If no display object is read from the current display status record, or if none of the read display objects are associated with an action entry, then the current display status record is excluded. If the display object and the action entry associated with the display object are read from the current display status record, then each action entry and the object carrier identifier associated with each action entry in the current display status record are read. Read subsequent display bearing status records whose corresponding bearing sequence position is later than the current display bearing status record but not later than the target bearing sequence position; If an action entry that is identical to each action entry in the current display carrier status record can be read in the same subsequent display carrier status record, and each identical action entry read is associated with an object carrier identifier, then the current display carrier status record is excluded. After comparing each display bearer status record, the display bearer status record with the latest corresponding acceptance time sequence position among the non-excluded display bearer status records is determined as the valid display bearer status record corresponding to the target acceptance time sequence position.

[0009] A further technical improvement of the present invention is that the method for determining the screen trigger object and forming the screen trigger succession relationship in step three includes: Read the sequence number, screen trigger subject identifier, and screen trigger position of the interactive acceptance event with screen trigger attribute, and read the valid display carrying status record corresponding to the sequence number. Read the display objects, the display carrier positions corresponding to each display object, and the action entry points corresponding to each display object from the valid display carrier status record. Compare the screen trigger positions with the display carrier positions corresponding to each display object and form the matched display objects based on the comparison results. When there is only one displayed object that is hit, the action entry and the preset short voice acceptance action corresponding to the hit displayed object are read, and the action entry and the preset short voice acceptance action are compared to form a range of acceptable actions based on the preset short voice acceptance actions that correspond to the action entry. When the number of hit display objects is one and the range of actions that can be accepted is not empty, the hit display object is identified as the screen trigger object, and the screen trigger object, the acceptance sequence position, the screen trigger subject identifier and the range of actions that can be accepted are associated and written to form a screen trigger acceptance relationship. When the number of displayed objects hit is not one, no screen trigger succession relationship is formed; When the number of displayed objects that are hit is one and the range of actions that can be received is empty, no screen triggering relationship is formed.

[0010] A further technical improvement of the present invention lies in: the method for generating the subject identifier used in the subject acceptance consistency determination in step five includes: Read the unified subject differentiation benchmark, which is used to map screen-triggered subject sampling and sound-emitting subject sampling to subject identifiers in the same subject identifier set; According to the unified subject differentiation benchmark, the screen trigger subject used to form the screen trigger succession relationship is sampled and mapped to the first subject identifier in the same subject identifier set, and the first subject identifier is used as the screen trigger subject identifier. Based on the same unified subject differentiation benchmark, the vocal subject used to form short speech succession relationship is sampled and mapped to the second subject identifier in the same subject identifier set, and the second subject identifier is used as the speech subject identifier; When both the screen trigger subject identifier and the voice subject identifier are formed, the screen trigger subject identifier and the voice subject identifier will be used as the basis for determining the consistency of subject inheritance. If neither the screen-triggered subject sampling nor the sound-generating subject sampling corresponds to a subject identifier in the same subject identifier set, no basis for determining subject continuity will be formed.

[0011] A further technical improvement of the present invention is that the method for determining the consistency of acceptance in step five includes: Read the screen trigger object, the sequence number of the sequence, and the range of actions that can be inherited in the screen trigger sequence relationship; and read the voice sequence number of the short voice sequence relationship, as well as the action expression content used to form the short voice sequence relationship. Compare the consistency of the object carrier identifier associated with the screen trigger object in the display carrier status record corresponding to the screen trigger object when the screen trigger connection relationship is formed with the object carrier identifier of the corresponding display object in the valid display carrier status record corresponding to the voice connection sequence position. Compare and match the content of the action with the range of actions that can be followed. Compare whether the object carrying identifier and action entry of the corresponding display object are continuous and consistent in the display carrying status record between the carrying sequence position of the screen trigger carrying relationship and the voice carrying sequence position. When the object carrier identifier is consistent, the action expression content matches the range of actions that can be accepted, and the object carrier identifier and the action entry point are continuously consistent, a result is formed that the consistency of acceptance is satisfied. If any of the following conditions are not met: the object carrier identifier is consistent, the action expression content matches the range of actions that can be accepted, and the object carrier identifier and the action entry are continuously consistent, a display acceptance inconsistency result is generated, and an acceptance execution blocking result is generated based on the display acceptance inconsistency result.

[0012] On the other hand, the present invention also discloses a multimodal collaborative instruction parsing and execution system for speakers with screens, characterized in that the system adopts the method described in any one of the above.

[0013] Compared with the prior art, the present invention has the following beneficial effects: This invention establishes interactive acceptance events and their acceptance sequence chains on the local end, enabling screen-triggered input, voice input, and display status to be associated and processed under the same acceptance sequence. Therefore, short voice input no longer generates execution objects in isolation based on voice content, but is matched with the previous screen-triggered object based on a clear acceptance sequence relationship, which helps improve the accuracy of instruction acceptance of the screen-equipped speaker in continuous multi-round interactions. Furthermore, this invention records the continuity of the display object's carrying status by display carrying status recording and object carrying identifier, so that the carrying status of the display object can be distinguished during the process of interface maintenance, refresh, or action entry change; therefore, when short voice depends on display object completion, it can be determined whether the screen trigger object still maintains a carrying state when voice input occurs based on the display carrying consistency, reducing erroneous execution caused by interface changes, object replacement, or action entry changes; On the other hand, by incorporating the screen trigger subject identifier and the voice subject identifier into the acceptance consistency determination, the present invention ensures that short voice acceptance is not only constrained by the display object and the action entry point, but also by the consistency of the input subject. Therefore, in a multi-person or multi-subject interactive environment, the probability of one subject's voice input being accepted by another subject's screen trigger object can be reduced, thereby improving the security and reliability of the multimodal collaborative instruction execution process of the screen-equipped speaker. Attached Figure Description

[0014] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0015] Figure 1 This is a schematic diagram of the method logic of the present invention. Detailed Implementation

[0016] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0017] Please see Figure 1 As shown, this invention provides a multimodal cooperative instruction parsing and execution method for speakers with screens, including: Step 1: Group the interactive response inputs generated by the local terminal of the speaker with screen into interactive acceptance events according to the order in which they enter the local interactive processing flow, generate acceptance sequence positions for the interactive acceptance events, and form an acceptance sequence chain. Specifically, in this embodiment, when the local terminal of the speaker with screen performs step one, it reads the processing entry points and their corresponding processing entry point identifiers pre-configured during local terminal initialization. The processing entry points include those for receiving screen-triggered input, those for receiving voice input, those for forming display carrying relationships, those for parsing voice content, and those for forming processing results corresponding to action expression content. The processing entry point identifiers are preset dimensionless numerical identifiers or character identifiers, and the identifiers of different processing entry points within the same interactive processing flow are different. Once configured, the processing entry point identifier remains unchanged during the local interactive processing flow, regardless of the specific content of the interactive response input, the order of entry, or the formation order of stage data. When any processing entry point forms stage data, the local terminal of the speaker with screen writes the processing entry point identifier corresponding to that processing entry point into the formed stage data, enabling the stage data to be mapped to the processing entry point through which it was formed via the processing entry point identifier.

[0018] When an interactive response input enters the local interactive processing flow, the local terminal of the speaker with a screen writes a trigger source identifier for that interactive response input according to the order in which they enter the local interactive processing flow. The trigger source identifier is a dimensionless identifier that can distinguish different interactive response inputs. When the same interactive response input continues to be processed through multiple processing entry points, the subsequent processing entry points use the trigger source identifier already written for that interactive response input; different interactive response inputs are written with different trigger source identifiers. Therefore, the stage data formed by the same interactive response input at different processing entry points are written with the same trigger source identifier, and the stage data formed by different interactive response inputs are written with different trigger source identifiers.

[0019] When any processing entry point forms stage data, the local terminal of the speaker with screen reads the maximum position value among the stage formation positions already written into the stage data of all processing entries during the current stage data merging process. If no written stage formation position is read, the stage formation position of the currently formed stage data is written as 1. When the maximum position value is read, the maximum position value is incremented by 1 and then written into the currently formed stage data as the stage formation position of that stage data. The stage formation position is a dimensionless ordinal value used to indicate the order in which stage data is formed in the local interactive processing flow. The stage formation position is uniformly incremented across all processing entries, and is not counted separately according to the processing entry type, so that stage data formed through different processing entries can be sorted and compared under the same position benchmark.

[0020] After each stage of data is formed, the local terminal of the speaker with screen reads the processing entry identifier, trigger source identifier, and stage formation position in the data of that stage. If the processing entry identifier, trigger source identifier, and stage formation position can be read in the data of that stage, the data of that stage is determined to be eligible for merging. If none of the processing entry identifier, trigger source identifier, and stage formation position can be read in the data of that stage, the data of that stage will not be written into the merging sequence and will not enter the subsequent merging process.

[0021] The speaker with a screen reads the entry sequence record locally. This entry sequence record is pre-written during local initialization and remains unchanged during the local interactive processing flow. The entry sequence record is configured according to the reading and formation relationships between processing entries in the local interactive processing flow. When the stage data formed by the previous processing entry is used as the reading object of the next processing entry, the identifiers of the previous and next processing entries are written to the entry sequence record in pairs, forming an adjacent entry relationship between them. Two processing entry identifiers that are not written in pairs in the entry sequence record are not considered to have an adjacent entry relationship.

[0022] The local terminal of the speaker with a screen reads the stage data that can participate in the merging process, and reads the processing entry identifier, trigger source identifier, and stage formation position from each stage data. The local terminal of the speaker with a screen groups stage data with the same trigger source identifier into the same merging sequence, and groups stage data with different trigger source identifiers into different merging sequences. After the merging sequence is formed, the local terminal of the speaker with a screen reads the stage formation position of each stage data in the same merging sequence, and arranges the stage data in the same merging sequence from first to last according to the stage formation position.

[0023] The local terminal of the speaker with a screen reads the first-ranked stage data in the same sequence to be merged, and uses this first-ranked stage data as the starting stage data for an interactive event. After the starting stage data is determined, the local terminal of the speaker with a screen reads two adjacent stages in the same sequence to be merged along the sorting direction, reads the processing entry identifiers corresponding to each of the two stages, and compares the two processing entry identifiers with the entry acceptance order record; at the same time, it reads the stage formation position corresponding to each of the two stages and compares whether the stage formation position corresponding to the later stage data is later than the stage formation position corresponding to the earlier stage data.

[0024] When the processing entry identifiers corresponding to two stages of data have an adjacent entry relationship, and the stage formation position corresponding to the later stage data is later than the stage formation position corresponding to the earlier stage data, the local end of the speaker with screen will write the later stage data into the interactive acceptance event to which the earlier stage data belongs, and continue to read the next set of adjacent stage data along the sorting direction for comparison.

[0025] When the processing entry identifiers corresponding to two stages of data do not have an adjacent succession relationship, or when the stage formation position corresponding to the later stage of data is not later than the stage formation position corresponding to the earlier stage of data, the local end of the speaker with screen does not write the later stage of data into the interactive succession event to which the earlier stage of data belongs. Instead, it uses the later stage of data as the starting stage of a new interactive succession event and continues to read the next set of adjacent stage data along the sorting direction for comparison until there are no uncompared adjacent stage data in the same sequence to be merged.

[0026] After each interactive event is generated, the local terminal of the speaker with a screen reads the processing entry identifiers corresponding to the data of each stage in the interactive event. If there is a processing entry identifier corresponding to a processing entry for receiving screen-triggered input, a screen trigger attribute is written to the interactive event; if there is a processing entry identifier corresponding to a processing entry for receiving voice input, a voice input attribute is written to the interactive event. Interactive events with screen trigger attributes are used as interactive events with screen trigger attributes in subsequent steps, and interactive events with voice input attributes are used as interactive events with voice input attributes in subsequent steps.

[0027] The local terminal of the speaker with a screen reads the stage data already written in each interactive acceptance event and reads the stage formation position corresponding to each stage data. For each interactive acceptance event, the stage data with the earliest stage formation position is used as the basis for generating the acceptance sequence position of that interactive acceptance event. This basis for generating the acceptance sequence position ensures that the temporal position of the interactive acceptance event corresponds to its earliest formed stage data, and is not affected by the formation time of subsequent stage data in the same interactive acceptance event.

[0028] The local terminal of the speaker with a screen reads multiple interactive succession events whose succession sequence positions have been determined, and reads the pre-written preset initial position value and preset succession interval value. The preset initial position value represents the succession sequence position of the first interactive succession event in the succession sequence chain, and the preset succession interval value represents the step size between the succession sequence positions of adjacent interactive succession events in the succession sequence chain; both the preset initial position value and the preset succession interval value are dimensionless ordinal values, and the preset succession interval value is greater than zero.

[0029] When the local end of the speaker with screen sorts each interactive event, it only reads the stage formation position corresponding to the generation basis of the acceptance sequence position of the interactive event as the sorting basis, and arranges multiple interactive events from first to last according to the sorting basis. The local end of the speaker with screen does not read the stage formation position corresponding to the stage data other than the generation basis of the acceptance sequence position in the same interactive event to participate in the sorting, nor does it use the physical timestamp of the interactive event as the sorting basis.

[0030] The local terminal of the speaker with a screen determines the acceptance sequence position of the first interactive event as a preset initial position value. Starting from the interactive events after the first one, the local terminal of the speaker with a screen sequentially reads the acceptance sequence position of the previous interactive event and adds the acceptance sequence position of the previous interactive event to the preset acceptance interval value to determine the acceptance sequence position of the current interactive event.

[0031] The local terminal of the speaker with a screen writes the sequence order of each interactive event into the event sequence chain, and also writes the sequential relationship between adjacent interactive events into the event sequence chain. After the event sequence chain is formed, it serves as the timing basis for subsequent steps to associate display carrying relationships, read interactive events with screen trigger attributes, and read interactive events with voice input attributes.

[0032] Step 2: Based on the correspondence between the displayed object and the action entry point, form a display carrying relationship, associate the display carrying relationship with the corresponding receiving sequence position and write it into the receiving sequence chain, and form a display carrying status record; based on the target receiving sequence position, determine the valid display carrying status record in the display carrying status record; Specifically, in step two, the local terminal of the speaker with a screen reads the succession sequence chain formed in step one, and sequentially reads the current succession sequence position according to the succession sequence position in the succession sequence chain from first to last. For the current succession sequence position, the local terminal of the speaker with a screen reads the display content corresponding to the current succession sequence position. When the current succession sequence position is associated with a newly generated display cache, the local terminal of the speaker with a screen reads the display content in the newly generated display cache; when the current succession sequence position is not associated with a newly generated display cache, the local terminal of the speaker with a screen reads the display content in the display cache whose succession sequence position is earlier than the current succession sequence position and is closest to the current succession sequence position, as the display content to be displayed under the current succession sequence position. The display content includes the display object and the action entry point called after the display object is triggered. The display object is the object in the local display interface that can be triggered by the screen; the action entry is the local action entry that is called after the display object is triggered. The action entry is used to represent the executable action corresponding to the display object and is not used as the processing entry for data merging in step one.

[0033] When reading the display content, the local terminal of the speaker with a screen sequentially reads the display object, the display object content field, the action entry point, the action entry point identifier, and the display carrying position. The display object content field is written when the display content is generated and stores the object name, object type, and interface hierarchy position of the display object. This is used to distinguish different display objects within the same sequential display relationship, but it is not used as an identifier to track the continuity of display object carrying between different sequential positions. The continuity of display object carrying between different sequential positions is determined by comparing the current display object and its action entry point with the display objects and their action entry points in the previous display carrying state record, and then by using and regenerating the object carrying identifier. The action entry point identifier is written during action entry configuration and is used to identify the action entry point called after the display object is triggered. The action entry point identifier and the processing entry point identifier in step one are configured independently. The action entry point identifier is used to identify the local action entry point called by the display object, while the processing entry point identifier is used to identify the processing entry point through which the stage data is formed in the local interactive processing flow. These two are identifiers at different levels.

[0034] The display carrying position is determined by the area boundary of the display object in the local display interface, and the area boundary is formed based on the pixel coordinates of the local display interface. The origin of the pixel coordinates is the upper left pixel position of the local display interface, and both the horizontal and vertical coordinate values ​​are measured in pixels. The display carrying position stores the boundary coordinates of the area where the display object is located. The local end of the speaker with screen associates and writes the display object, the display object content field, the action entry, the action entry identifier, and the display carrying position to form the display carrying relationship corresponding to the current acceptance sequence position. When no display object is read in the display content corresponding to the current acceptance sequence position, a display carrying relationship without the display object is formed. When a display object is read but no action entry associated with the display object is read, the display object, the display object content field, and the display carrying position are written into the display carrying relationship, but the action entry and action entry identifier corresponding to the display object are not written. When a display object is associated with multiple action entries, the display object and each action entry are associated and written into the display carrying relationship respectively; the same display object corresponds to one object carrying identifier under the same acceptance sequence position, and each action entry identifier is stored as a set of action entry identifiers associated with the display object. When comparing action entry points that are identical, the comparison will be based on whether the action entry point identifier set contains the corresponding action entry point identifier.

[0035] After the display carrying relationship is formed, the local terminal of the speaker with screen will associate the display carrying relationship with the current receiving sequence position and write it into the receiving sequence chain. During the writing process, the interactive receiving events, receiving sequence positions, and sequential receiving relationships already formed in the receiving sequence chain will not be changed; the display carrying relationship will be written as the display carrying content corresponding to the current receiving sequence position, which will be used to form the display carrying status record corresponding to the current receiving sequence position.

[0036] When the current acceptance sequence position is the first acceptance sequence position in the acceptance sequence chain, the local terminal of the speaker with screen reads the display carrying relationship corresponding to the current acceptance sequence position. When a display object is read in this display carrying relationship, the local terminal of the speaker with screen reads each display object sequentially and generates an object carrying identifier for each display object. The object carrying identifier is a dimensionless ordinal value used to distinguish the carrying continuity of the display object in different acceptance sequence positions. Since there is no previous display carrying status record before the first acceptance sequence position, there is no object carrying identifier that can be used for each display object corresponding to the first acceptance sequence position. When the local terminal of the speaker with screen generates a new object carrying identifier for each display object, it is allocated sequentially according to the number of display objects that have completed the writing of object carrying identifiers under the current acceptance sequence position; the object carrying identifier of the first display object that has completed the writing of object carrying identifiers is 1, and the object carrying identifier of subsequent display objects is incremented by 1 based on the previous allocation value. After the object carrier identifier of each display object is determined, the local terminal of the speaker with screen writes the first and second acceptance sequence positions, the display carrier relationship, and the object carrier identifier of each display object together to form a display carrier status record corresponding to the first and second acceptance sequence positions. If no display object is read in the display carrier relationship corresponding to the first and second acceptance sequence positions, the first and second acceptance sequence positions and the display carrier relationship are written together to form a display carrier status record without display objects.

[0037] When the current succession sequence position is not the first succession sequence position in the succession sequence chain, the local terminal of the speaker with screen reads the display carrying relationship corresponding to the current succession sequence position, as well as the prior display carrying state record that is adjacent to the current succession sequence position and whose succession sequence position is earlier than the current succession sequence position. The prior display carrying state record is the display carrying state record formed corresponding to the previous succession sequence position, which is used to provide a comparison basis for the inheritance and regeneration of the object carrying identifier of the current display object. When no display object is read in the display carrying relationship corresponding to the current succession sequence position, the current succession sequence position and the display carrying relationship are associated and written to form a display carrying state record without a display object.

[0038] When a display object is read from the display carrying relationship corresponding to the current receiving sequence position, the local end of the speaker with screen sequentially reads each display object in the display carrying relationship, and takes the currently read display object as the current display object. It then reads the display object content field, display carrying position, action entry point, and action entry point identifier corresponding to the current display object. The local end of the speaker with screen compares the current display object and its associated action entry point identifier set with the display objects and their associated action entry point identifier sets in the previous display carrying state record. Display object consistency is determined by the consistency of the display object content field and the display carrying position; action entry consistency is determined by the consistency of the action entry point identifier set associated with the current display object with the action entry point identifier set associated with the corresponding display object in the previous display carrying state record.

[0039] Condition A1 is that a corresponding display object exists in the prior display status record that is identical to the current display object and has the same action entry point. When condition A1 is met, the local terminal of the speaker with screen reads the object carrier identifier from the prior display status record of the corresponding display object and uses this object carrier identifier as the object carrier identifier of the current display object. Therefore, when neither the display object nor its action entry point has changed, the current display object uses the object carrier identifier from the prior display status record.

[0040] Condition A2 is that there is no corresponding display object in the prior display carrying status record that matches the current display object. When condition A2 is met, the local terminal of the speaker with screen generates a new object carrying identifier for the current display object. Condition A3 is that there is a corresponding display object in the prior display carrying status record that matches the current display object, and the action entry of the corresponding display object is different from the action entry of the current display object. When condition A3 is met, the local terminal of the speaker with screen generates a new object carrying identifier for the current display object. When generating a new object carrying identifier, the local terminal of the speaker with screen sequentially reads the object carrying identifiers saved in the already formed display carrying status record, and reads the object carrying identifier corresponding to the display object whose object carrying identifier has been written under the current carrying sequence position. If no written object carrying identifier is read, the object carrying identifier of the current display object is written as 1; if a written object carrying identifier is read, the largest object carrying identifier is incremented by 1 and written to the current display object as the new object carrying identifier. The new object carrying identifier is not duplicated with the already written object carrying identifier, so that the current display object does not use the object carrying identifier in the prior display carrying status record. For display objects that exist in the prior display bearing status record but are not read in the current display bearing relationship, their object bearing identifiers are not written into the display bearing status record corresponding to the current bearing sequence position.

[0041] After determining the object carrier identifier for each display object in the display carrier relationship, the local terminal of the speaker with screen writes the current acceptance sequence position, the display carrier relationship corresponding to the current acceptance sequence position, and the object carrier identifier of each display object together to form a display carrier status record corresponding to the current acceptance sequence position. After the display carrier status record is formed, it saves the current acceptance sequence position, display object, display object content field, action entry, action entry identifier, display carrier position, and object carrier identifier, which are used to determine the valid display carrier status record based on the target acceptance sequence position later. Once a display carrier status record has been formed for the same acceptance sequence position, a new display carrier status record will not be formed again for that acceptance sequence position.

[0042] When it is necessary to determine the valid display carrying status record based on the target acceptance timing position, the local terminal of the speaker with screen reads the target acceptance timing position. The target acceptance timing position is determined by the acceptance timing position corresponding to the interactive acceptance event for which the display carrying status needs to be determined. The local terminal of the speaker with screen reads the already formed display carrying status records, and reads the display carrying status records whose corresponding acceptance timing position is not later than the target acceptance timing position from the already formed display carrying status records, and includes the read display carrying status records in the current candidate records; display carrying status records whose corresponding acceptance timing position is later than the target acceptance timing position do not participate in the determination of the valid display carrying status record in this time.

[0043] The local terminal of the speaker with a screen reads the candidate records one by one in order of their corresponding acceptance sequence positions, and uses the currently read display acceptance status record as the current display acceptance status record. If no display object is read in the current display acceptance status record, the current display acceptance status record is excluded from the current candidate records. If a display object is read in the current display acceptance status record, but none of the read display objects are associated with an action entry point, the current display acceptance status record is excluded from the current candidate records. Exclusion is only used to determine the valid display acceptance status record corresponding to the current target acceptance sequence position; it does not delete already formed display acceptance status records. Excluded display acceptance status records can still be reread and compared as display acceptance status records with corresponding acceptance sequence positions no later than the subsequent target acceptance sequence positions during the determination of valid display acceptance status records for other target acceptance sequence positions.

[0044] When a display object and its associated action entry are read from the current display carrying status record, the local end of the speaker with screen reads all action entries associated with all display objects in the current display carrying status record, and reads the object carrying identifier associated with each action entry. When there are multiple display objects in the current display carrying status record, all action entries associated with all display objects are compared; when the same display object is associated with multiple action entries in the current display carrying status record, all of those action entries are compared. The local end of the speaker with screen continues to read subsequent display carrying status records whose corresponding succession timing position is later than the current display carrying status record but not later than the target succession timing position. Subsequent display carrying status records are used to determine whether the action entries in the current display carrying status record have been completely covered by subsequent succession timing positions.

[0045] The local terminal of the speaker with a screen compares each action entry in the current display carrying status record with the action entries in the same subsequent display carrying status record; if the action entries are the same, the determination is based on the consistency of the action entry identifier. Condition B1 is that an action entry identical to each action entry in the current display carrying status record can be read in the same subsequent display carrying status record, and each read identical action entry is associated with an object carrying identifier. If condition B1 is met, the current display carrying status record is excluded from the current candidate records. Condition B2 is that no same subsequent display carrying status record satisfies condition B1. If condition B2 is met, the current display carrying status record is not excluded due to this comparison.

[0046] After the local end of the speaker with screen completes the comparison of each display bearer status record with a corresponding acceptance time sequence position no later than the target acceptance time sequence position, it reads the display bearer status records that have not been excluded from the current candidate records. Condition C1 is that a display bearer status record that has not been excluded is read. When condition C1 is met, the acceptance time sequence positions corresponding to the non-excluded display bearer status records are compared, and the display bearer status record with the latest corresponding acceptance time sequence position is determined as the valid display bearer status record corresponding to the target acceptance time sequence position. Since a new display bearer status record is not repeatedly formed for the same acceptance time sequence position after it has been formed, each acceptance time sequence position corresponds to at most one display bearer status record; based on this, the display bearer status record with the latest corresponding acceptance time sequence position among the non-excluded display bearer status records is a unique record. Condition C2 is that no non-excluded display bearer status record is read. When condition C2 is met, no valid display bearing status record corresponding to the target bearing sequence position is formed; when subsequent steps read the valid display bearing status record according to the target bearing sequence position, the reading result is empty.

[0047] Step 3: Compare the interactive acceptance events with screen trigger attributes with the valid display acceptance status records corresponding to their acceptance sequence positions to determine the screen trigger object and form a screen trigger acceptance relationship that includes the screen trigger object, the acceptance sequence position, and the screen trigger subject identifier. Specifically, in step three, the local end of the speaker with a screen reads the succession sequence chain formed in step one, and reads the interactive succession events that have been written with screen trigger attributes from the succession sequence chain, treating them as interactive succession events with screen trigger attributes. The local end of the speaker with a screen reads the succession sequence position corresponding to the interactive succession event, and reads the screen trigger position and screen trigger subject identifier saved with the screen trigger input in the interactive succession event. The screen trigger position is written when the screen trigger input forms stage data, including horizontal trigger coordinate values ​​and vertical trigger coordinate values; both horizontal and vertical trigger coordinate values ​​take the upper left corner pixel position of the local end display interface as the origin and are measured in pixels.

[0048] The screen trigger subject identifier is written during the screen trigger input formation stage. When writing the screen trigger subject identifier, the local terminal of the speaker with screen reads the screen trigger subject sample corresponding to the screen trigger input and, according to the pre-written unified subject differentiation benchmark, maps the screen trigger subject sample to the first subject identifier in the same subject identifier set, using this first subject identifier as the screen trigger subject identifier. The screen trigger subject sample saves the trigger sample identifier corresponding to the screen trigger input, and the voice subject sample saves the voice sample identifier corresponding to the voice input. The unified subject differentiation benchmark saves the correspondence between the trigger sample identifier, the voice sample identifier, and the subject identifiers in the same subject identifier set, used to map the screen trigger subject sample and the voice subject sample to the subject identifiers in the same subject identifier set, respectively. This step only reads the screen trigger subject identifier and uses it to form the screen trigger continuity relationship; it does not perform subject continuity consistency judgment.

[0049] The local terminal of the speaker with a screen reads the valid display carrying status record formed in step two according to the receiving sequence position of the interactive receiving event with screen trigger attributes. The valid display carrying status record is the display carrying status record determined for this receiving sequence position; after being read in this step, it serves as the comparison basis for determining the screen trigger object and does not change due to subsequent changes in the displayed content. If no valid display carrying status record is read, no display object hit comparison is performed, and the number of hit display objects is recorded as zero. If no screen trigger subject identifier is read, no screen trigger receiving relationship is formed. If no horizontal trigger coordinate value is read, no display object hit comparison is performed, and the number of hit display objects is recorded as zero. If no vertical trigger coordinate value is read, no display object hit comparison is performed, and the number of hit display objects is recorded as zero.

[0050] Upon receiving a valid display carrying status record, screen trigger subject identifier, horizontal trigger coordinate value, and vertical trigger coordinate value, the local terminal of the speaker with screen reads the display object, display object content field, display carrying position, action entry point, and action entry point identifier from the valid display carrying status record. The display carrying position and the screen trigger position use the same pixel coordinate origin and unit of measurement. The horizontal boundary coordinates in the display carrying position include the left boundary coordinates and the right boundary coordinates, and the vertical boundary coordinates include the upper boundary coordinates and the lower boundary coordinates.

[0051] The local terminal of the speaker with a screen sequentially reads each display object from the valid display carrying status record and takes the currently read display object as the current display object to be compared. The local terminal reads the display carrying position corresponding to the current display object to be compared, compares the horizontal trigger coordinate value with the left and right boundary coordinates, and compares the vertical trigger coordinate value with the top and bottom boundary coordinates. If the horizontal trigger coordinate value is not less than the left boundary coordinate and less than the right boundary coordinate, and the vertical trigger coordinate value is not less than the top boundary coordinate and less than the bottom boundary coordinate, the current display object to be compared is determined as the matched display object. If the above horizontal and vertical range conditions are not met, the current display object to be compared is not determined as the matched display object. The above coordinate comparison uses a half-open interval convention in pixel coordinates. The left and top boundary coordinates belong to the display object area range, while the right and bottom boundary coordinates do not, to avoid adjacent display objects being simultaneously matched at shared boundaries. After completing the comparison of each display object in the valid display carrying status record, the number of matched display objects is counted.

[0052] When only one display object is matched, the local terminal of the speaker with screen reads all action entries corresponding to the matched display object and the action entry identifiers corresponding to each action entry, and reads the preset short voice response actions. The preset short voice response actions are written to the configuration file during local terminal initialization; each preset short voice response action saves the short voice action name, short voice action identifier, and corresponding action entry identifier. The short voice action name is used as the matching object for subsequent action content; the corresponding action entry identifier is written to the local terminal during initialization according to the short voice actions that the action entry can handle, and is used for comparison with the action entry identifier corresponding to the matched display object. The local terminal of the speaker with screen compares each action entry identifier corresponding to the matched display object item by item with the corresponding action entry identifier saved in the preset short voice response actions, and writes the preset short voice response actions with corresponding relationships into the range of acceptable actions; when multiple action entry identifiers correspond to the same preset short voice response action, that preset short voice response action is written only once within the range of acceptable actions.

[0053] The local terminal of the speaker with screen determines whether a screen-triggered connection is formed based on the number of displayed objects hit and the range of actions that can be received. The determination conditions include: Condition D1 is that the number of displayed objects hit is one, and at least one preset short voice action is read within the range of actions that can be received; Condition D2 is that the number of displayed objects hit is zero; Condition D3 is that the number of displayed objects hit is greater than one; Condition D4 is that the number of displayed objects hit is one, and no preset short voice action is read within the range of actions that can be received.

[0054] When condition D1 is met, the local terminal of the speaker with a screen identifies the hit display object as the screen trigger object, and associates and writes the screen trigger object, the sequence number of the interactive event with screen trigger attributes, the screen trigger subject identifier, and the scope of actions that can be taken, thus forming a screen trigger acceptance relationship. After the screen trigger acceptance relationship is formed, it serves as the basis for subsequent steps to read the screen trigger object, the sequence number of the event, the screen trigger subject identifier, and the scope of actions that can be taken.

[0055] When condition D2 is met, the local terminal of the speaker with a screen does not create a screen trigger object, nor does it establish a screen trigger succession relationship; when condition D3 is met, the local terminal of the speaker with a screen does not create a screen trigger object, nor does it establish a screen trigger succession relationship; when condition D4 is met, the local terminal of the speaker with a screen does not create a screen trigger object, nor does it establish a screen trigger succession relationship. For interactive succession events that meet conditions D2, D3, and D4, subsequent steps reading the screen trigger succession relationship according to the succession sequence will result in an empty result.

[0056] Step 4: Analyze the acceptance requirements of interactive acceptance events with voice input attributes; when the content expressed by the object needs to be completed by the display object and the content expressed by the action belongs to the preset short voice acceptance action, the acceptance sequence position of the interactive acceptance event is used as the voice acceptance sequence position, and a short voice acceptance relationship containing the voice acceptance sequence position and the voice subject identifier is formed. Specifically, in step four, the local end of the speaker with a screen reads the succession sequence chain formed in step one, and reads the interactive succession events that have been written with voice input attributes from the succession sequence chain, treating them as interactive succession events with voice input attributes. The local end of the speaker with a screen reads the succession sequence position corresponding to this interactive succession event, and reads the voice input text and speaker sample saved with the voice input formation stage data in this interactive succession event. If no voice input text is read, no short voice succession relationship is formed; when subsequent steps read short voice succession relationships according to this succession sequence position, the reading result is empty.

[0057] When the voice input text is read, the local end of the speaker with screen reads the acceptance requirement parsing benchmark. The acceptance requirement parsing benchmark is written during local initialization, saving object field labels, action field labels, independent object expression benchmarks, display object completion markers, and preset short voice acceptance actions. The independent object expression benchmark saves the object name, object category, and object alias that can directly point to the operated object; the display object completion marker is a marker value written during local initialization, used to mark the object expression content that needs to rely on display object completion in the voice parsing result; the preset short voice acceptance action saves the short voice action name, short voice action identifier, and corresponding action entry identifier. The preset short voice acceptance action read in step four is the same data in the same configuration file as the preset short voice acceptance action used in step three to form the range of acceptable actions. The short voice action name and short voice action identifier use the same naming and encoding rules in steps three and four; this step reads the short voice action name and short voice action identifier.

[0058] The local terminal of the speaker with a screen performs local semantic parsing on the voice input text, generating a voice parsing result. The voice parsing result saves object field labels, action field labels, the field content corresponding to the field labels, and object completion marker fields. When the field content corresponding to an object field label is empty, the local terminal of the speaker with a screen writes the display object completion marker into the object completion marker field. When the field content corresponding to an object field label is not empty, the local terminal of the speaker with a screen compares the field content with the independent object representation benchmark item by item; if the comparison result does not match the independent object representation benchmark, the display object completion marker is written into the object completion marker field. The local terminal of the speaker with a screen reads the object representation content according to the object field labels and reads the action representation content according to the action field labels. The object representation content is used to represent the content in the voice input text that points to the object being operated on, and the action representation content is used to represent the content in the voice input text that points to the action to be performed.

[0059] The local end of the speaker with a screen reads the object representation content and object completion flag field from the speech parsing result and forms an object completion status field. If no object representation content is read, the object completion status field is set to "needs completion"; if object representation content is read but does not match the independent object representation baseline, the object completion status field is set to "needs completion"; if object representation content is read and the object completion flag field contains a display object completion flag, the object completion status field is set to "needs completion"; if object representation content is read, matches the independent object representation baseline, and the object completion flag field does not contain a display object completion flag, the object completion status field is set to "no completion required". The object completion status field indicates whether the current voice input requires display object completion of the operated object.

[0060] The local terminal of the speaker with a screen compares the action expression content with the names of the preset short voice action sequences item by item. If no action expression content is found, an action sequence status record is created, and the action sequence status field is written as "missed". If action expression content is found, the local terminal compares the action expression content with each short voice action name item by item. If a short voice action name matching the action expression content is found during the item-by-item comparison, an action sequence status record is created, the action sequence status field is written as "hit", and the corresponding short voice action identifier is written to the action sequence status record. If no short voice action name matching the action expression content is found after the item-by-item comparison, an action sequence status record is created, and the action sequence status field is written as "missed". The action sequence status record is used to indicate whether the action expression content in the current voice input belongs to the preset short voice action sequence and is not used as the result of command execution.

[0061] The local terminal of the speaker with a screen reads the same unified subject differentiation benchmark used in step three, and according to this benchmark, maps the sound-emitting subject sample to the second subject identifier in the same subject identifier set, using this second subject identifier as the voice subject identifier. The same subject identifier set is the same as the subject identifier set corresponding to the screen-triggered subject sample in step three, and the second subject identifier and the first subject identifier in step three reside in the same identifier space. If no sound-emitting subject sample is read, no voice subject identifier is formed; if the sound-emitting subject sample is not mapped to the second subject identifier in the same subject identifier set, no voice subject identifier is formed. This step only reads the voice subject identifier and uses it to form short voice continuity relationships; it does not perform subject continuity consistency determination.

[0062] The local terminal of the speaker with a screen reads the object completion status field, action acceptance status record, and voice subject identifier formation status. When the object completion status field indicates "needs completion," the action acceptance status field indicates "hit," and a voice subject identifier has been formed, the local terminal of the speaker with a screen uses the acceptance sequence position of the interactive acceptance event with voice input attributes as the voice acceptance sequence position. It then associates and writes the voice acceptance sequence position, voice subject identifier, action expression content, object completion status field, and action acceptance status record to form a short voice acceptance relationship. The action expression content is the field content extracted from the voice input text according to the action field label and saved in the form of the original field content from the voice parsing result. The action acceptance status record saves the action acceptance status field and the short voice action identifier, ensuring that the action expression content corresponds to its hit preset short voice acceptance action. After the short voice acceptance relationship is formed, it serves as the basis for subsequent steps to read the voice acceptance sequence position, voice subject identifier, action expression content, object completion status field, and action acceptance status record.

[0063] When the object completion status field indicates that completion is not required, the local end of the speaker with screen does not form a short voice succession relationship; when the action succession status field indicates that it has not been hit, the local end of the speaker with screen does not form a short voice succession relationship; when no voice subject identifier is formed, the local end of the speaker with screen does not form a short voice succession relationship. When no short voice succession relationship is formed, subsequent steps read the short voice succession relationship according to the succession sequence position, and the reading result is empty.

[0064] Step 5: Call the valid display carrying status record according to the voice reception sequence; determine the consistency of display reception and the consistency of main body reception based on the valid display carrying status record, screen trigger reception relationship and short voice reception relationship, and form the instruction execution result or reception execution blocking result according to the judgment result; Specifically, in step five, the local end of the speaker with a screen reads the succession sequence chain formed in step one, and reads the interactive succession events that have been written into the voice input attributes according to the succession sequence position in the succession sequence chain from first to last. The succession sequence position of the currently read interactive succession event is used as the succession sequence position for this processing. The local end of the speaker with a screen reads the short voice succession relationship formed in step four according to the succession sequence position for this processing. If no short voice succession relationship corresponding to the succession sequence position for this processing is read, the display succession consistency judgment and the main body succession consistency judgment are not executed, and a succession execution blocking result is formed. The succession execution blocking result saves the blocking reason that the succession sequence position for this processing and the short voice succession relationship are read as empty, and ends the processing in step five corresponding to the succession sequence position for this processing. In this step, after the succession execution blocking result is formed, the local end of the speaker with a screen will no longer continue to execute the subsequent display succession consistency judgment, main body succession consistency judgment, and instruction execution result formation.

[0065] When a short voice sequence relationship is read, the local end of the speaker with a screen reads the voice sequence position, voice subject identifier, action expression content, object completion status field, and action sequence status record from the short voice sequence relationship. The action sequence status record stores the action sequence status field and short voice action identifier formed in step four, used for verification against the action sequence matching result formed in this step. The local end of the speaker with a screen compares the voice sequence position with the sequence position of the current processing. If they are inconsistent, a sequence execution blocking result is formed, and step five processing corresponding to the current sequence position ends. If they are consistent, the object completion status field is read and compared with the field that needs completion. If the object completion status field does not need completion, a sequence execution blocking result is formed, and step five processing corresponding to the current sequence position ends. If the object completion status field needs completion, the valid display carrying status record formed in step two is called according to the voice sequence position, and the called valid display carrying status record is used as the valid display carrying status record corresponding to the voice sequence position. If no valid display carrying status record corresponding to the voice receiving sequence position is invoked, a display receiving inconsistency result is generated, and a receiving execution blocking result is generated based on the display receiving inconsistency result, thus ending the processing of step five corresponding to the receiving sequence position.

[0066] The local terminal of the speaker with a screen reads the screen trigger sequence relationships formed in step three, and reads screen trigger sequence relationships whose sequence position is earlier than the voice sequence position from the already formed screen trigger sequence relationships. If no screen trigger sequence relationship whose sequence position is earlier than the voice sequence position is read, a display sequence inconsistency result is generated, and a sequence execution blocking result is generated based on the display sequence inconsistency result, ending the processing in step five corresponding to the sequence position. When a screen trigger sequence relationship whose sequence position is earlier than the voice sequence position is read, for each screen trigger sequence relationship, the voice sequence position is subtracted from the sequence position of the screen trigger sequence relationship to form a sequence position difference; the screen trigger sequence relationship with the smallest sequence position difference is read. When the number of screen trigger acceptance relationships with the smallest acceptance position difference is one, this screen trigger acceptance relationship is taken as the current screen trigger acceptance relationship; when the number of screen trigger acceptance relationships with the smallest acceptance position difference is greater than one, a display acceptance inconsistency result is formed, and an acceptance execution blocking result is formed based on the display acceptance inconsistency result, ending the processing of step five corresponding to the acceptance timing position.

[0067] When the current screen trigger connection relationship is read, the local terminal of the speaker with screen reads the screen trigger object, connection sequence position, screen trigger subject identifier, and range of actions that can be connected within the current screen trigger connection relationship. The local terminal of the speaker with screen reads the valid display carrying status record corresponding to the connection sequence position in the current screen trigger connection relationship, using it as the display carrying status record corresponding to the formation of the screen trigger object. If the display carrying status record is not read, a display connection inconsistency result is generated, and a connection execution blocking result is generated based on the display connection inconsistency result, ending the processing of step five corresponding to the connection sequence position. When the display carrying status record is read, the local terminal of the speaker with screen reads the display object content field, object carrying identifier, and action entry identifier set associated with the screen trigger object from the display carrying status record, and uses the object carrying identifier as the screen-side object carrying identifier.

[0068] The local terminal of the speaker with a screen reads the action expression content and the range of actions that can be accepted. The action expression content is the field content extracted from the voice input text in step four according to the action field tags and saved in the short voice succession relationship. The range of actions that can be accepted saves the preset short voice succession actions written in step three. The local terminal of the speaker with a screen compares the action expression content with the short voice action names of each preset short voice succession action in the range of actions that can be accepted, and counts the number of short voice action names that match the action expression content. When there is only one short voice action name that matches the action expression content, an action succession matching result is formed, and the short voice action identifier and the corresponding action entry identifier in the preset short voice succession action are written into the action succession matching result, where the corresponding action entry identifier serves as the matching action entry identifier. When there is zero short voice action names that match the action expression content, a display succession inconsistency result is formed, and a succession execution blocking result is formed based on the display succession inconsistency result, ending the processing in step five corresponding to the succession sequence position. When the number of short speech action names that are consistent with the content of the action expression is greater than one, a display inconsistency result is generated, and a transmission execution blocking result is generated based on the display inconsistency result, thus ending the processing of step five corresponding to the transmission sequence position.

[0069] After the action acceptance matching result is formed, the local end of the speaker with screen compares the short voice action identifier in the action acceptance matching result with the short voice action identifier in the action acceptance status record. If they match, the action acceptance matching result is retained; if they do not match, a display acceptance inconsistency result is generated, and an acceptance execution blocking result is generated based on the display acceptance inconsistency result, thus ending the processing of step five corresponding to the acceptance timing position.

[0070] After the action acceptance matching result is retained, the local terminal of the speaker with screen reads the display object with the same display object content field as the screen trigger object and associated with the matching action entry identifier from the valid display bearer status record corresponding to the voice acceptance sequence position, and uses the read display object as the candidate voice side corresponding display object. If no candidate voice side corresponding display object is read, a display acceptance inconsistency result is formed, and an acceptance execution blocking result is formed based on the display acceptance inconsistency result, ending the processing of step five corresponding to the acceptance sequence position. When a candidate voice side corresponding display object is read, the local terminal of the speaker with screen reads the object bearer identifier of the candidate voice side corresponding display object one by one, and compares the consistency of each object bearer identifier with the screen side object bearer identifier. When the number of candidate voice-side display objects whose object bearer identifier matches the screen-side object bearer identifier is one, the candidate voice-side display object is determined as the voice-side display object, and a result of consistent object bearer identifier is formed. When the number of candidate voice-side display objects whose object bearer identifier matches the screen-side object bearer identifier is zero, a result of inconsistent display acceptance is formed, and an acceptance execution blocking result is formed based on the result of inconsistent display acceptance, ending the fifth step of the current processing corresponding to the acceptance timing position. When the number of candidate voice-side display objects whose object bearer identifier matches the screen-side object bearer identifier is greater than one, a result of inconsistent display acceptance is formed, and an acceptance execution blocking result is formed based on the result of inconsistent display acceptance, ending the fifth step of the current processing corresponding to the acceptance timing position.

[0071] The local terminal of the speaker with a screen reads the display carrying status records whose corresponding receiving sequence position is no earlier than the receiving sequence position of the current screen trigger receiving relationship and no later than the voice receiving sequence position, and reads them one by one in order of receiving sequence position from first to last. The display carrying status records are the display carrying status records already formed in step two, and are not limited to valid display carrying status records. The reading range includes the display carrying status records corresponding to the receiving sequence position of the current screen trigger receiving relationship and the display carrying status records corresponding to the voice receiving sequence position. For the currently read display carrying status records, the local terminal of the speaker with a screen reads the display objects that have the same display object content fields as the screen trigger object and are associated with a matching action entry identifier, and uses the read display objects as candidate continuous comparison display objects. If no candidate continuous comparison display objects are read, a display receiving inconsistency result is formed, and a receiving execution blocking result is formed based on the display receiving inconsistency result, ending the processing of step five corresponding to the receiving sequence position. When a candidate continuous comparison display object is read, the object carrier identifier of each candidate continuous comparison display object is read one by one, and each object carrier identifier is compared with the screen-side object carrier identifier. If the number of candidate continuous comparison display objects whose object carrier identifier matches the screen-side object carrier identifier in the currently read display carrier status record is one, then this candidate continuous comparison display object is included as a continuous comparison display object in the current display carrier status record; if the number of candidate continuous comparison display objects whose object carrier identifier matches the screen-side object carrier identifier in the currently read display carrier status record is zero, or the number is greater than one, then a display acceptance inconsistency result is formed, and an acceptance execution blocking result is formed based on the display acceptance inconsistency result, ending the processing of step five corresponding to the acceptance timing position of this processing. When a unique continuous comparison display object is determined in each display carrier status record, a continuous consistency result of object carrier identifier and action entry is formed.

[0072] When consistent results have been achieved in object carrier identifier, action acceptance matching, and continuous consistency between object carrier identifier and action entry, the local terminal of the speaker with screen generates a display acceptance consistency satisfaction result. When an inconsistent display acceptance result has been generated, the local terminal of the speaker with screen generates an acceptance execution blocking result based on the inconsistent display acceptance result. The acceptance execution blocking result saves the acceptance timing position of this processing, the voice acceptance timing position, the acceptance timing position of the current screen triggered acceptance relationship, and the reason for the inconsistent display acceptance, and ends the processing step five corresponding to the current acceptance timing position.

[0073] The local terminal of the speaker with screen reads the same unified subject differentiation benchmark used in step three, and reads the screen trigger subject identifier in the current screen trigger succession relationship and the voice subject identifier in the short voice succession relationship. The screen trigger subject identifier is the first subject identifier in the same subject identifier set corresponding to the screen trigger subject sample according to the unified subject differentiation benchmark, and the voice subject identifier is the second subject identifier in the same subject identifier set corresponding to the voice subject sample according to the same unified subject differentiation benchmark. When both the screen trigger subject identifier and the voice subject identifier are read, they are used as the basis for determining subject succession consistency; when the screen trigger subject identifier is not read, no basis for determining subject succession consistency is formed, and a succession execution blocking result is formed, ending the processing of step five corresponding to the succession timing position of this processing; when the voice subject identifier is not read, no basis for determining subject succession consistency is formed, and a succession execution blocking result is formed, ending the processing of step five corresponding to the succession timing position of this processing.

[0074] The local terminal of the speaker with a screen compares the screen trigger subject identifier with the voice subject identifier for consistency. If they match, a subject acceptance consistency satisfaction result is formed; if they do not match, a subject acceptance inconsistency result is formed, and an acceptance execution blocking result is generated based on the subject acceptance inconsistency result, ending the processing step five corresponding to the acceptance timing position of this processing. When both the display acceptance consistency satisfaction result and the subject acceptance consistency satisfaction result have been formed, the local terminal of the speaker with a screen reads the screen trigger object, the matching action entry identifier, the action expression content, and the voice acceptance timing position, and reads the action entry associated with the screen trigger object and whose action entry identifier is the matching action entry identifier from the display carrying state record corresponding to the formation of the screen trigger object, forming an instruction execution result. The instruction execution result saves the current processing acceptance timing position, screen trigger object, voice acceptance timing position, action entry, matching action entry identifier, action expression content, screen trigger subject identifier, and voice subject identifier.

[0075] In another embodiment, the present invention also provides a multimodal cooperative instruction parsing and execution system for speakers with screens, comprising: The receiving module is used to read the interactive response input generated by the local terminal of the speaker with screen, read the processing entry identifier, trigger source identifier, and stage formation position; classify stage data with the same trigger source identifier into the same merge sequence, and merge the stage data into interactive receiving events according to the entry receiving order and stage formation position; read the stage data with the earliest stage formation position in each interactive receiving event as the basis for generating the receiving time sequence position, generate the receiving time sequence position according to the sorting result between events, and form a receiving time sequence chain.

[0076] The bearer module is used to read the display content corresponding to each bearing sequence position according to the bearing sequence chain, read the display object, display object content field, display bearer position, action entry and action entry identifier, and form a display bearer relationship; associate the display bearer relationship with the bearing sequence position and write it into the bearing sequence chain; determine whether to reuse or regenerate the object bearer identifier based on the display object content field, display bearer position and action entry identifier set, and form a display bearer status record; and determine the valid display bearer status record from the display bearer status record according to the target bearing sequence position.

[0077] The trigger module is used to read interactive acceptance events with screen trigger attributes, including their acceptance sequence position, screen trigger location, and screen trigger subject identifier. It reads valid display carrying status records according to the acceptance sequence position, compares the screen trigger location with the display carrying locations of each display object, and when only one display object is hit, reads the action entry identifier corresponding to the hit display object and compares it with the corresponding action entry identifier in the preset short voice acceptance actions to form a range of acceptable actions. When only one display object is hit and the range of acceptable actions is not empty, it determines the screen trigger object and forms a screen trigger acceptance relationship.

[0078] The voice module is used to read interactive events with voice input attributes, read the voice input text and the speaker sample; form a voice parsing result based on the parsing benchmark of the acceptance requirements, read the object expression content, action expression content and object completion mark field to form an object completion status field; compare the action expression content with the preset short voice acceptance action to form an action acceptance status record; form a voice subject identifier according to a unified subject differentiation benchmark, and form a short voice acceptance relationship when the object completion status field is "needs completion", the action acceptance status field is "hit" and the voice subject identifier has been formed.

[0079] The execution module is used to call the valid display carrying status record according to the voice carrying sequence position in the short voice carrying relationship, read the screen trigger carrying relationship with the smallest difference in the carrying position, and form the current screen trigger carrying relationship; form a matching action entry identifier according to the action expression content and the range of actions that can be carried, and determine the consistency of display carrying based on the display object content field, the matching action entry identifier, and the object carrying identifier; read the screen trigger subject identifier and the voice subject identifier to determine the consistency of subject carrying; and form an instruction execution result when both display carrying consistency and subject carrying consistency are satisfied, otherwise form a carrying execution blocking result and stop the formation of the instruction execution result corresponding to the carrying sequence position of this processing.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for parsing and executing multimodal cooperative instructions for speakers with screens, characterized in that, include: Step 1: Group the interactive response inputs generated by the local terminal of the speaker with screen into interactive acceptance events according to the order in which they enter the local interactive processing flow, generate acceptance sequence positions for the interactive acceptance events, and form an acceptance sequence chain. Step 2: Based on the correspondence between the displayed object and the action entry, form a display carrying relationship, associate the display carrying relationship with the corresponding receiving sequence position and write it into the receiving sequence chain, and form a display carrying status record; Based on the target's acceptance sequence position, determine the valid display bearing status record in the display bearing status record; Step 3: Compare the interactive acceptance events with screen trigger attributes with the valid display acceptance status records corresponding to their acceptance sequence positions to determine the screen trigger object and form a screen trigger acceptance relationship that includes the screen trigger object, the acceptance sequence position, and the screen trigger subject identifier. Step 4: Analyze the acceptance requirements of interactive acceptance events with voice input attributes; when the content expressed by the object needs to be completed by the display object and the content expressed by the action belongs to the preset short voice acceptance action, the acceptance sequence position of the interactive acceptance event is used as the voice acceptance sequence position, and a short voice acceptance relationship containing the voice acceptance sequence position and the voice subject identifier is formed. Step 5: Call the valid display carrying status record according to the voice receiving sequence position; based on the valid display carrying status record, screen trigger receiving relationship and short voice receiving relationship, determine the display receiving consistency and the main receiving consistency, and form the instruction execution result or receiving execution blocking result according to the determination result.

2. The multimodal cooperative instruction parsing and execution method for speakers with screens according to claim 1, characterized in that, The method for determining the basis for generating the succession sequence position in step one includes: Read the stage data formed by the interactive response input through different processing entry points in the local interactive processing flow, and obtain the processing entry identifier, trigger source identifier and stage formation position corresponding to each stage data; Read the entry acceptance sequence record, which stores the adjacent acceptance relationship between different processing entry identifiers; Stage data with the same trigger source identifier are grouped into the same merge sequence, and the stage data in the same merge sequence are sorted according to the stage formation position. The first-ranked stage data in the same sequence to be merged is used as the starting stage data for an interactive event. Read two adjacent stages of data in the same sequence to be merged along the sorting direction, and compare the processing entry identifiers corresponding to the two stages with the entry acceptance order records; When the processing entry identifiers corresponding to two stages of data have an adjacent entry relationship, and the stage formation position corresponding to the later stage data is later than the stage formation position corresponding to the earlier stage data, the later stage data is written into the interaction acceptance event to which the earlier stage data belongs. When the processing entry identifiers corresponding to two stages of data do not have an adjacent succession relationship, or when the stage formation position corresponding to the later stage data is not later than the stage formation position corresponding to the earlier stage data, the later stage data will be used as the starting stage data of the new interactive succession event. The earliest stage data in the same interactive event is used as the basis for generating the timing sequence of the interactive event.

3. The multimodal cooperative instruction parsing and execution method for speakers with screens according to claim 2, characterized in that, Step one, which generates succession sequence positions based on succession sequence position generation criteria and writes them into the succession sequence chain, includes: Read multiple interactive acceptance events whose acceptance sequence positions have been determined as the basis for generation, and read the preset initial position value and preset acceptance interval value. The preset acceptance interval value is greater than zero. For each interactive acceptance event, the ordering of events is based solely on the order of the stages corresponding to the acceptance sequence of that interactive acceptance event. Multiple interactive events are arranged in order of priority based on the sequence of events; The order of the first interactive event is set as the preset initial position value. Starting from the first interactive event in the sequence, read the sequence position of the previous interactive event in sequence, and add the sequence position of the previous interactive event to the preset interval value to determine the sequence position of the current interactive event. Write the sequence order of each interactive event into the sequence chain, and write the sequential relationship between adjacent interactive events into the sequence chain.

4. The multimodal collaborative instruction parsing and execution method for speakers with screens according to claim 1, characterized in that, The method for generating the display bearing status record in step two includes: When the current acceptance sequence position is not the first acceptance sequence position in the acceptance sequence chain, read the display carrying relationship corresponding to the current acceptance sequence position, as well as the previous display carrying status record that is adjacent to the current acceptance sequence position and whose acceptance sequence position is earlier than the current acceptance sequence position. Read each display object in the display carrying relationship in sequence, and take the currently read display object as the current display object, and read the action entry corresponding to the current display object; Compare the currently displayed object and its corresponding action entry with the displayed object and action entry in the previously displayed carrying state record; When there is a corresponding display object in the prior display status record that is identical to the current display object and has the same action entry point, the object carrier identifier of the corresponding display object in the prior display status record is used as the object carrier identifier of the current display object. When there is no corresponding display object in the prior display status record that matches the current display object, or when there is a corresponding display object that matches the current display object but the action entry is different, a new object carrier identifier is generated for the current display object. After determining the object carrier identifier for each display object in the display carrier relationship, the current acceptance sequence position, display carrier relationship, and object carrier identifier of each display object are associated and written to form a display carrier status record corresponding to the current acceptance sequence position.

5. The multimodal cooperative instruction parsing and execution method for speakers with screens according to claim 4, characterized in that, The method for determining valid display bearer status records in step two includes: Read the target acceptance sequence position, and read the display acceptance status record whose corresponding acceptance sequence position is not later than the target acceptance sequence position; Read the display bearer status records one by one, and use the currently read display bearer status record as the current display bearer status record; If no display object is read from the current display status record, or if none of the read display objects are associated with an action entry, then the current display status record is excluded. If the display object and the action entry associated with the display object are read from the current display status record, then each action entry and the object carrier identifier associated with each action entry in the current display status record are read. Read subsequent display bearing status records whose corresponding bearing sequence position is later than the current display bearing status record but not later than the target bearing sequence position; If an action entry that is identical to each action entry in the current display carrier status record can be read in the same subsequent display carrier status record, and each identical action entry read is associated with an object carrier identifier, then the current display carrier status record is excluded. After comparing each display bearer status record, the display bearer status record with the latest corresponding acceptance time sequence position among the non-excluded display bearer status records is determined as the valid display bearer status record corresponding to the target acceptance time sequence position.

6. The multimodal cooperative instruction parsing and execution method for speakers with screens according to claim 5, characterized in that, The method for determining the screen trigger object and establishing the screen trigger succession relationship in step three includes: Read the sequence number, screen trigger subject identifier, and screen trigger position of the interactive acceptance event with screen trigger attribute, and read the valid display carrying status record corresponding to the sequence number. Read the display objects, the display carrier positions corresponding to each display object, and the action entry points corresponding to each display object from the valid display carrier status record. Compare the screen trigger positions with the display carrier positions corresponding to each display object and form the matched display objects based on the comparison results. When there is only one displayed object that is hit, the action entry and the preset short voice acceptance action corresponding to the hit displayed object are read, and the action entry and the preset short voice acceptance action are compared to form a range of acceptable actions based on the preset short voice acceptance actions that correspond to the action entry. When the number of hit display objects is one and the range of actions that can be accepted is not empty, the hit display object is identified as the screen trigger object, and the screen trigger object, the acceptance sequence position, the screen trigger subject identifier and the range of actions that can be accepted are associated and written to form a screen trigger acceptance relationship. When the number of displayed objects hit is not one, no screen trigger succession relationship is formed; When the number of displayed objects that are hit is one and the range of actions that can be received is empty, no screen triggering relationship is formed.

7. The multimodal cooperative instruction parsing and execution method for speakers with screens according to claim 1, characterized in that, The method for generating the subject identifier used in step five for determining subject consistency includes: Read the unified subject differentiation benchmark, which is used to map screen-triggered subject sampling and sound-emitting subject sampling to subject identifiers in the same subject identifier set; According to the unified subject differentiation benchmark, the screen trigger subject used to form the screen trigger succession relationship is sampled and mapped to the first subject identifier in the same subject identifier set, and the first subject identifier is used as the screen trigger subject identifier. Based on the same unified subject differentiation benchmark, the vocal subject used to form short speech succession relationship is sampled and mapped to the second subject identifier in the same subject identifier set, and the second subject identifier is used as the speech subject identifier; When both the screen trigger subject identifier and the voice subject identifier are formed, the screen trigger subject identifier and the voice subject identifier will be used as the basis for determining the consistency of subject inheritance. If neither the screen-triggered subject sampling nor the sound-generating subject sampling corresponds to a subject identifier in the same subject identifier set, no basis for determining subject continuity will be formed.

8. The multimodal cooperative instruction parsing and execution method for speakers with screens according to claim 6, characterized in that, Step five shows the methods for determining consistency, including: Read the screen trigger object, the sequence number of the sequence, and the range of actions that can be inherited in the screen trigger sequence relationship; and read the voice sequence number of the short voice sequence relationship, as well as the action expression content used to form the short voice sequence relationship. Compare the consistency of the object carrier identifier associated with the screen trigger object in the display carrier status record corresponding to the screen trigger object when the screen trigger connection relationship is formed with the object carrier identifier of the corresponding display object in the valid display carrier status record corresponding to the voice connection sequence position. Compare and match the content of the action with the range of actions that can be followed. Compare whether the object carrying identifier and action entry of the corresponding display object are continuous and consistent in the display carrying status record between the carrying sequence position of the screen trigger carrying relationship and the voice carrying sequence position. When the object carrier identifier is consistent, the action expression content matches the range of actions that can be accepted, and the object carrier identifier and the action entry point are continuously consistent, a result is formed that the consistency of acceptance is satisfied. If any of the following conditions are not met: the object carrier identifier is consistent, the action expression content matches the range of actions that can be accepted, and the object carrier identifier and the action entry are continuously consistent, a display acceptance inconsistency result is generated, and an acceptance execution blocking result is generated based on the display acceptance inconsistency result.

9. A multimodal collaborative instruction parsing and execution system for speakers with screens, characterized in that, The system employs the method described in any one of claims 1-8.

Images