A multimedia interactive visual control system based on intelligent speech recognition

Abstract

The application discloses a multimedia interactive visual control system based on intelligent voice recognition and relates to the technical field of intelligent voice processing. The system collects continuous voice and obtains recognition candidates and start and end times, and synchronously establishes a visual state record containing display change time, visual objects and corresponding relations; corresponding visual states are determined according to the voice time range of target description content, target visual objects are determined in combination with visual objects and display attributes, and current corresponding objects are determined along the object corresponding relations in continuous visual states, so that visual control instructions are formed to change display states, and the visual state record is updated. The application can keep the consistency of the control objects pointed by voice when the multimedia visual state continuously changes, and reduce the object miscontrol caused by the time dislocation of voice and picture change.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice processing technology, and more specifically, to a multimedia interactive visual control system based on intelligent voice recognition. Background Technology

[0002] Existing multimedia interactive visual control systems based on intelligent speech recognition typically first perform complete recognition and semantic analysis of the continuous speech input by the user, and then determine the object to be controlled based on the recognition results and the current visual state at the time of speech recognition completion. In dynamic visual interactive scenarios such as exhibition halls, interactive screens, and cultural tourism displays, multimedia images often change continuously according to the playback sequence, and may also be affected by other interactive events such as touch and motion sensing. Therefore, within the time range from the start of input to the completion of recognition of a continuous speech, the displayed image may have undergone one or more state transitions. For speech commands containing object referential content or visual attribute descriptions, the semantic content that bears the role of target referentiality actually corresponds to the visual object that the user is facing when the relevant speech segment is formed. However, existing systems usually perform target matching based on the visual state after the entire speech is recognized. When the visual state at the time of target semantic formation is inconsistent with the visual state at the time of speech recognition completion, even if the speech content and control intent can be correctly recognized, the control action may be associated with other visual objects that appear after the state change, or with the changed position and performance state of the original target object, thus causing a mismatch between the speech recognition result and the actual controlled object.

[0003] Therefore, the problem lies in how to accurately determine the actual visual object pointed to by continuous speech during the continuous change of multimedia visual state, and maintain the consistency of the controlled object after the visual state continues to change, so as to avoid mis-control of the object when there is a time misalignment between speech recognition and image changes. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, this invention provides a multimedia interactive visual control system based on intelligent voice recognition. By establishing a temporal correlation between voice content and corresponding visual states, and combining the correspondence between visual objects in different visual states, the current corresponding object is determined, thereby reducing misidentification and miscontrol of objects caused by the asynchronous nature of the voice recognition process and screen changes.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A multimedia interactive visual control system based on intelligent voice recognition includes a voice acquisition module, a multimedia visual output terminal, and a processor, wherein the processor performs the following processing: The system collects continuous speech input from the user, performs speech recognition on the continuous speech, obtains multiple recognition candidates arranged in the order of speech, and records the start and end times of each recognition candidate in the continuous speech; it also records the time when the multimedia visual output terminal displays changes, the visual objects before and after the display changes, and the correspondence between the same visual object before and after the display changes, thus obtaining a visual state record. Based on the semantic relationship between each recognition candidate, the target description content describing the object to be controlled and the corresponding control action are determined from continuous speech, and the speech time range corresponding to the target description content is determined based on the start and end times of the recognition candidates that make up the target description content. The speech time range is compared with the display change time in the visual state record to determine one or more visual states corresponding to the speech time range, and the visual objects and display attributes in one or more visual states are read. When there are multiple recognition candidates at the same speech position, each recognition candidate is matched with the visual objects and display attributes in the corresponding visual state. The recognition result of the target description content is determined according to the recognition candidate that can form a corresponding relationship with the same visual object, and the visual object corresponding to the recognition result is determined as the target visual object. The current corresponding object is determined based on the relationship between the visual state of the target visual object and the current visual state; when the target visual object is in the current visual state, the target visual object is determined as the current corresponding object; when the visual state of the target visual object is different from the current visual state, the correspondence between the target visual object in adjacent visual states is read sequentially according to the display change order in the visual state record to determine the current corresponding object of the target visual object in the current visual state. Based on the control action and the corresponding object, a visual control instruction is generated, and the multimedia visual output terminal is controlled to change the display state of the corresponding object according to the visual control instruction.

[0006] The technical effects and advantages of this invention are as follows: 1. The audio time range of the target description content is matched with the display change time, and the correspondence between visual objects in continuous visual states is tracked to the current visual state, so that the object pointed to by the continuous audio can remain consistent after the screen changes continuously, reducing the miscontrol of objects caused by the misalignment of audio time and screen changes. 2. Multiple recognition candidates are retained at the same speech position, and each recognition candidate is matched with the visual object and display attribute in the corresponding visual state. The actual visual state can be used to further filter different recognition candidates, reducing the ambiguity caused by relying solely on speech recognition results. 3. The visual state record continuously saves the display change time, the visual objects before and after the change and the corresponding relationship, so that the target visual object still has a traceable state evolution path after being replaced, generated or the display state is changed, providing continuous data basis for determining the current corresponding object across visual states. 4. When a visual object corresponds to multiple result visual objects, further distinguish them by combining the recognition content in the target description; if it still cannot be uniquely determined, stop generating visual control instructions to avoid uncertain objects from continuing to enter control execution; 5. After each visual control is completed, the time of the display change, the visual objects before and after the execution, the display attributes and their corresponding relationships are rewritten into the visual state record. This ensures that the new screen changes caused by the control action are incorporated into the subsequent processing in a timely manner, and that subsequent voice recognition, target object tracking and re-control are all based on the updated actual visual state. Attached Figure Description

[0007] Figure 1 This is a system processing flowchart of the present invention; Figure 2 This is a flowchart of the visual state recording formation and recognition process corresponding to visual states in this invention. Figure 3 A flowchart illustrating the process of identifying the target description content and determining the target visual object in this invention; Figure 4 This is a flowchart illustrating the current object determination, visual control, and visual state recording update processes of this invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] like Figure 1As shown, this invention provides a multimedia interactive visual control system based on intelligent speech recognition, including a speech acquisition module, a multimedia visual output terminal, and a processor. During system operation, the processor acquires continuous speech input by the user, performs speech recognition on the continuous speech, obtains multiple recognition candidates arranged in the order of the speech, and records the start and end times of each recognition candidate in the continuous speech. Simultaneously, it records the time of display changes at the multimedia visual output terminal, the visual objects before and after the display changes, and the correspondence between the same visual object before and after the display changes, thus obtaining a visual state record. The processor determines action candidates and corresponding control actions based on the action content corresponding to each recognition candidate and the display control operation, and then... The semantic relationship between action candidates and other speech location recognition candidates determines the target description content describing the object of the controlled action. Then, the speech time range is determined based on the start and end times of the recognition candidates that make up the target description content. Based on the speech time range and the visual state record, one or more corresponding visual states are determined. The recognition candidates are matched with the visual objects and display attributes in the corresponding visual states to determine the recognition result of the target description content and the target visual object. Based on the relationship between the visual state where the target visual object is located and the current visual state, the current corresponding object is determined. Based on the control action and the current corresponding object, a visual control command is formed to control the multimedia visual output terminal to change the display state of the current corresponding object.

[0010] This embodiment provides a detailed explanation of the following steps: formation of recognition candidates, determination of the start and end times of recognition candidates, formation of visual state records, determination of visual object correspondence, determination of control actions, determination of target description content, determination of speech time range, correspondence between recognition candidates and visual states, determination of recognition results of target description content, determination of target visual objects, determination of the current corresponding object, formation of visual control instructions and change of the display state of the current corresponding object, and updating of visual state records.

[0011] I. Identifying Candidate Formation The voice acquisition module acquires continuous voice input from the user, and the processor performs voice recognition on the audio data according to the acquisition order of the continuous voice. The voice acquisition module continuously acquires continuous speech during the user's speech and converts the continuous speech into corresponding audio data according to the acquisition sequence. The processor receives the audio data and performs speech recognition while maintaining the original acquisition sequence, so that the audio data acquired first enters the corresponding speech recognition processing first, and the audio data acquired later continues to be processed according to the acquisition sequence. The processor determines the corresponding speech position based on the sequential position of the audio data in the continuous speech and records the recognition content obtained by speech recognition at each speech position, so that the formed recognition candidates are consistent with the actual input sequence of the continuous speech.

[0012] For recognition results with different recognition content at the same speech position, the processor retains them as recognition candidates, resulting in multiple recognition candidates arranged in the order of speech. When speech recognition yields two or more different recognition results for the same speech location, the processor retains each recognition result and uses them as recognition candidates for the corresponding speech location. During the recognition candidate formation stage, other recognition results are not deleted based on one of the recognition results. When only one recognition result is obtained for the same speech location, the recognition result is used as a recognition candidate for the corresponding speech location. The recognition candidates are arranged according to the order of the speech locations in continuous speech, so that different recognition results existing at the same speech location are retained until the determination of the target description content and the determination of the recognition result in combination with the visual state.

[0013] The processor associates each recognition candidate at the same speech location with the corresponding audio data; When the processor saves each recognition candidate, it records the continuous audio frames corresponding to the recognition content of the corresponding voice position formed during the speech recognition process as the audio data corresponding to that voice position, and at the same time records the association relationship between the recognition candidate and the corresponding voice position and audio data. When there are multiple recognition candidates for the same voice position, each recognition candidate is associated with the audio data corresponding to that voice position. Thus, the processor can read the first and last audio frames in the corresponding audio data according to the recognition candidate, and continue to determine the start and end times of the recognition candidate.

[0014] II. Identification of Candidate Start and End Times The processor reads the first and last audio frames from the audio data associated with each candidate for recognition; Based on the established association between the recognition candidates and the audio data, the processor locates the audio data corresponding to each recognition candidate and reads the first and last audio frames acquired in the order of audio data acquisition, which are respectively used as the first and last audio frames. The first and last audio frames define the time boundaries of the audio data associated with the corresponding recognition candidate in continuous speech.

[0015] The processor determines the start time of the corresponding recognition candidate based on the acquisition time of the first audio frame, and determines the end time of the corresponding recognition candidate based on the acquisition time of the last audio frame. When the voice acquisition module acquires continuous voice, each audio frame has a corresponding acquisition time according to the actual acquisition time. The processor reads the acquisition time of the first audio frame and records it as the start time of the corresponding recognition candidate, and reads the acquisition time of the last audio frame and records it as the end time of the corresponding recognition candidate. When there are multiple recognition candidates at the same voice position and each recognition candidate is associated with the same audio data, each recognition candidate obtains the start time and end time corresponding to the audio data. The start time and end time limit the actual time range of the corresponding recognition candidate in the continuous voice.

[0016] The processor determines the order of each speech position in continuous speech based on the acquisition time of the first audio frame in the audio data associated with each speech position; The processor reads the acquisition time of the first audio frame in the audio data associated with each speech position, and arranges each speech position in order of acquisition time from first to last. Speech positions with earlier first audio frame acquisition times are placed at the beginning of the arrangement order, and speech positions with later first audio frame acquisition times are placed at the end of the arrangement order, so that the arrangement order of each speech position is consistent with the actual acquisition order of continuous speech.

[0017] The processor records the recognition candidates and start and end times corresponding to each speech position in the order they are arranged. Based on the established arrangement order, the processor records each speech position, one or more corresponding recognition candidates, and the start and end times of each recognition candidate in the order of their positions in continuous speech. When there are multiple recognition candidates for the same speech position, the multiple recognition candidates are retained under the same speech position, and their corresponding start and end times are recorded respectively, so that the speech position, recognition candidate, and start and end time maintain a corresponding relationship.

[0018] III. Formation of Visual State Records like Figure 2 As shown in the figure, solid arrows represent processing flow or writing relationships, while dashed arrows represent reading visual state records when executing corresponding steps; the processor synchronously acquires display execution data sent to the multimedia visual output terminal; During the operation of the multimedia visual output terminal, the processor continuously acquires display execution data sent to the multimedia visual output terminal and records it according to the actual order in which the display execution data is sent. When the processor starts maintaining the visual state record, it reads the visual objects, object identifiers, object names, object types, and display attributes that currently exist in the multimedia visual output terminal and records the corresponding content as the initial visual state. During continuous voice acquisition and voice recognition, the processor continues to synchronously acquire display execution data. The display execution data corresponds to the display content and display change content currently being executed by the multimedia visual output terminal. The display change operation, visual object, object identifier, and display attributes are all obtained from the corresponding display execution data. The acquisition time of the audio frame and the display change time are recorded according to the same timing benchmark used by the processor, so that the start and end times of the recognition candidate, the voice time range, and the display change time in the visual state record can be directly compared according to time sequence.

[0019] The processor determines display change operations from the display execution data that cause the addition, deletion, replacement, or change of display attributes of visual objects; For the continuously acquired display execution data, the processor reads the execution content item by item and compares the visual objects and display attributes before and after execution. When the execution content causes a new visual object to appear in the multimedia visual output terminal, the corresponding execution content is identified as a display change operation that causes the addition of a visual object. When an existing visual object no longer exists after execution, the corresponding execution content is identified as a display change operation that causes the deletion of a visual object. When a visual object is replaced by another visual object after execution, the corresponding execution content is identified as a display change operation that causes the replacement of a visual object. When a visual object remains but its corresponding display attribute changes, the corresponding execution content is identified as a display change operation that causes the change of display attribute. Thus, the actual display change operation that causes the change in display content is determined from the display execution data.

[0020] The processor reads the visual object that has been changed by the display operation; For a determined display change operation, the processor reads the visual object that has been added, deleted, replaced, or whose display attributes have changed, based on the execution content corresponding to the corresponding display change operation in the display execution data, and simultaneously reads the object name, object type, and display attributes associated with the corresponding visual object. In this embodiment, a visual object refers to the display content in the multimedia visual output terminal display content that has a corresponding object identifier and can be determined independently by the display execution data. The object name represents the name content corresponding to the visual object, the object type represents the display content category to which the visual object belongs, and the display attribute represents the attribute content in the display execution data corresponding to the current display state of the visual object. The display attributes include the display position, display size, display color, and corresponding attributes in the visible / hidden state of the corresponding visual object. When a single display change operation acts on multiple visual objects simultaneously, the processor reads each visual object separately and maintains the correspondence between each visual object and the same display change operation. The object name, object type, and display attributes are recorded along with the corresponding visual object for comparison between the candidate recognition content and the visual object in the corresponding visual state.

[0021] The processor records the object identifier and display attributes of the visual object before the execution of the change operation; Before the actual execution of the display change operation, the processor reads the object identifier and display attributes of the visual object affected by the display change operation, and uses the corresponding values ​​as the recorded content before the display change. The object identifier is taken from the identifier information maintained by the multimedia visual output terminal for the corresponding visual object, and the display attributes are taken from the corresponding display attribute values ​​before the display change operation is performed on the corresponding visual object, thereby recording the state of the visual object before the display change occurs.

[0022] After the display change operation is completed, the processor reads the object identifier and display attributes of the corresponding visual object to obtain the visual object before and after the display change; When a display change operation is completed, the processor reads the object identifier and display attributes of the corresponding visual object again, and saves the reading results along with the object identifier and display attributes already recorded before execution. For display change operations that add visual objects, the object identifier and display attributes of the newly added visual object are added to the record after execution. For display change operations that delete visual objects, the object identifier and display attributes of the deleted visual object are recorded before execution. For display change operations that replace visual objects or change display attributes, the object identifier and display attributes of the corresponding visual objects before and after execution are saved respectively, thus obtaining the visual objects before and after a display change.

[0023] The processor determines the time when the change operation takes effect as the display change time. When a display change operation is actually applied to the multimedia visual output terminal and a corresponding display result is formed, the processor reads the effective time of the display change operation and records the effective time as the display change time corresponding to this display change. The display change time represents the time position when the multimedia visual output terminal enters the state after the display change from the state before the display change, and is recorded using the same timing reference as the acquisition time of the audio frame.

[0024] The processor records the visual objects before and after the display changes in chronological order, thus obtaining a visual state record; The processor uses the initial visual state as the starting state for visual state recording. For multiple consecutive display change operations, it records the display change time and the visual objects before and after each display change in chronological order. The visual state maintained by the multimedia visual output terminal between two adjacent display change times corresponds to the visual state maintained by the multimedia visual output terminal within the corresponding time period. The display result formed after the last display change time corresponds to the current continuous visual state. The visual objects before and after the display change time reflect the changes in visual objects in adjacent visual states, thus forming a visual state record that is continuously recorded in chronological order. When no display change has occurred since the initial visual state was formed, the initial visual state itself constitutes the current continuous visual state.

[0025] IV. Determining the Correspondence Between Visual Objects The processor reads the source visual object before the display change operation is executed and the result visual object after the display change operation is executed; Based on the visual objects before and after the display change obtained during the formation of the visual state record, the processor determines the visual object that was affected by the corresponding display change operation before the display change operation was executed as the source visual object, and determines the visual object corresponding to the source visual object after the display change operation was executed as the result visual object. The object identifier and display attributes of the source visual object are taken from the recorded content before the display change operation was executed, and the object identifier and display attributes of the result visual object are taken from the recorded content after the display change operation was executed. Both the source visual object and the result visual object are limited to corresponding processing before and after the same display change operation.

[0026] When the source visual object and the result visual object have the same object identifier, the processor establishes a correspondence between the source visual object and the result visual object based on the same object identifier. The processor reads and compares the object identifiers of the source visual object and the result visual object respectively. When the two object identifiers are the same, the source visual object before the display change operation is executed and the result visual object after the operation are executed are recorded as corresponding to each other. When the display attributes of the source visual object and the result visual object change, the same object identifier still indicates that the visual objects before and after the display change remain continuous at the object level.

[0027] When a display change operation replaces a source visual object with a result visual object that has a different object identifier, the processor establishes a correspondence between the source visual object and the result visual object based on the replacement relationship in the display execution data. When the source visual object and the result visual object have different object identifiers, the processor reads the display execution data that caused the current display change. When the display change operation recorded in the display execution data is to replace the source visual object with the result visual object, the processor further reads the source visual object that was replaced and the result visual object after replacement in the display execution data, and establishes a correspondence between the two according to the replacement relationship, so that the continuous correspondence between the source visual object and the result visual object is still preserved when the object identifier changes.

[0028] When a display change operation generates a result visual object with a different object identifier based on a source visual object, the processor establishes a correspondence between the source visual object and the result visual object based on the generation relationship in the display execution data. When the execution data shows that the resulting visual object is generated from the source visual object through a display change operation, the processor reads the source visual object and the generated resulting visual object from the execution data and establishes a correspondence between them based on the generation relationship. When a generation operation generates more than two resulting visual objects, the processor records the generation relationship between the source visual object and each resulting visual object, so that one source visual object can correspond to more than two resulting visual objects. When determining the current corresponding object, the processor further distinguishes multiple resulting visual objects by combining the target description content.

[0029] For visual objects that are not affected by display change operations and exist continuously before and after the display change, the processor establishes a correspondence between the display before and after the display change based on the same object identifier; In addition to the visual objects directly affected by the display change operation, the processor also compares other visual objects that exist continuously before and after the display change. For visual objects that exist before and after the display change and whose object identifiers remain the same, the processor establishes a correspondence between the visual objects before the display change and the visual objects after the display change, so that visual objects that are not affected by this display change operation continue to maintain a correspondence between adjacent visual states.

[0030] The processor writes the resulting correspondence and the corresponding display change time into the visual state record; The processor writes the correspondences formed by the same object identifiers, the replacement relationships in the display execution data, the generation relationships, and the correspondences of visual objects that are not affected by the display change operation, together with the corresponding display change time, into the visual state record, and associates each correspondence with the two visual states before and after the display change time; as display changes occur continuously, the visual state record forms the correspondences of adjacent visual states and visual objects that can be read sequentially according to the display change time.

[0031] V. Determining Control Actions The processor reads the display control operations supported by the multimedia visual output terminal; The processor reads the various display control operations supported by the multimedia visual output terminal and synchronously reads the corresponding action content of each display control operation. The display control operation represents the display changes that the multimedia visual output terminal can perform on the visual object, and the action content is taken from the operation name or command content corresponding to the display control operation.

[0032] The processor compares the recognition candidates for each voice position with the corresponding action content of the display control operation; The processor reads the recognition candidates according to the order of each speech position in the continuous speech, and reads the action content corresponding to the display control operation for each recognition candidate. The recognition content that can express the same display control operation is matched with the corresponding action content based on whether the action expressed by the recognition content of the recognition candidate is consistent with the action content corresponding to the display control operation. When there are multiple recognition candidates for the same speech position, each recognition candidate is compared separately.

[0033] For recognition candidates that correspond to display control operations, the processor will identify the corresponding recognition candidate as an action candidate; Based on the comparison results between the recognition candidates and the action content, the processor reads the recognition candidates that have been formed with the display control operation and records the corresponding recognition candidates as action candidates; the voice position and start and end time corresponding to the action candidate continue to use the content already recorded by the corresponding recognition candidate.

[0034] The processor determines the display control operation corresponding to the action candidate as the control action; The processor uses the established correspondence between action candidates and display control operations to read the display control operations corresponding to the action candidates and determine the corresponding display control operations as control actions; the control actions serve as the basis for subsequently determining the objects affected by the controlled actions and forming visual control instructions.

[0035] VI. Determining the Content of the Target Description The processor determines the recognition candidates describing the object of the controlled action based on the semantic relationship between the action candidates and the recognition candidates of other speech locations; The processor uses the speech position of the action candidate as a reference, reads the recognition candidates corresponding to the speech positions before and after the action candidate in the order of the continuous speech, and forms candidate sentences containing the action candidate and other recognition candidates. The processor distinguishes between action expression, object expression, and object description expression in each candidate sentence. The recognition content corresponding to the action candidate is taken as the action expression, the recognition content that can represent the name or type of the visual object affected by the control action is taken as the object expression, and the recognition content that limits the position, size, color, or visibility state of the object expression is taken as the object description expression. When a recognition candidate corresponds to an object expression or object description expression, and together with the action candidate in the speech order, constitutes the sentence content for the same control action, the corresponding recognition candidate is determined as the recognition candidate describing the object affected by the controlled action. When there are multiple recognition candidates at the same speech position, each recognition candidate performs the above determination, and continues to retain different recognition candidates that meet the conditions of object expression or object description expression.

[0036] The processor combines the identified recognition candidates according to the order of speech to obtain the target description content; The processor reads the speech positions corresponding to each recognition candidate describing the object of the controlled action, and arranges the speech positions from front to back according to the determined speech position arrangement order. For different speech positions, the corresponding recognition candidates are organized according to the original order in the continuous speech. When there are more than two recognition candidates for the same speech position, different recognition candidates are retained at the corresponding speech positions, and the different recognition contents of the same speech position are not connected continuously in the order of front and back. In this way, the target description content is obtained, and the original order of each speech position in the continuous speech and the different recognition candidates corresponding to the same speech position are maintained.

[0037] In cases where the same speech position in the target description corresponds to different recognition content, the processor retains the different recognition candidates corresponding to the same speech position in the target description. When a certain speech location involved in the target description generates two or more different recognition contents during the aforementioned speech recognition process, the processor continues to retain the different recognition candidates corresponding to that speech location in the target description content, and maintains the established association between each recognition candidate and the same speech location, corresponding audio data, and start and end time, so that different recognition candidates can continue to participate in the subsequent recognition result determination process based on visual state.

[0038] VII. Determining the Time Range of Speech The processor reads the start and end times of each identification candidate that makes up the target description content; For each recognition candidate that makes up the target description content, the processor reads them in the order of the speech positions in the target description content and obtains the start time and end time corresponding to each recognition candidate from the aforementioned recorded content; when multiple recognition candidates are retained for the same speech position, the start time and end time associated with each recognition candidate are read respectively.

[0039] The processor compares the start times of each identification candidate that makes up the target description content and determines the earliest start time as the start time of the target description content. The processor compares the start times of each recognition candidate that makes up the target description content in chronological order, and selects the earliest start time as the start time of the target description content, thereby determining the front-end time boundary of the target description content in continuous speech.

[0040] The processor compares the end times of each identification candidate that makes up the target description content and determines the latest end time of the target description content. For each candidate recognition, the processor compares the end times in chronological order and selects the latest end time as the end time of the target description content, thereby determining the back-end time boundary of the target description content in continuous speech.

[0041] The processor obtains the speech time range corresponding to the target description content based on the start and end times of the target description content; The processor uses the start time of the target description content as the starting point of the speech time range and the end time of the target description content as the ending point of the speech time range, and defines the continuous time interval between the two as the speech time range corresponding to the target description content; the speech time range represents the time interval actually occupied by the target description content in continuous speech.

[0042] 8. Identifying the correspondence between candidates and visual states like Figure 2 As shown, the processor compares the start and end times of the speech time range with the display change times in the visual state record to determine the display change times that fall within the speech time range. The processor reads the start and end times of the speech time range and sequentially reads each display change time according to the time sequence recorded in the visual state. For each display change time, it determines whether the display change time is no earlier than the start time and no later than the end time of the speech time range. Display change times that meet this condition are determined as display change times falling within the speech time range. The speech time range and display change times use the same timing reference, thus enabling the determination of the actual display changes that occur during the input of the target description content.

[0043] When there is no display change time within the voice time range, the processor determines the visual state within the voice time range as the corresponding visual state. If no display change time falls within the speech time range, the processor determines the visual state covering the corresponding speech time range based on the time position of the speech time range in the visual state record. If there is a recorded display change time before the start time of the speech time range, the processor reads the visual state formed after the most recent display change took effect before the start time and continuously covers the speech time range, and determines the corresponding visual state as the corresponding visual state. If there is no recorded display change time before the start time of the speech time range, the processor reads the initial visual state in the visual state record and determines it as the corresponding visual state when the initial visual state continuously covers the speech time range.

[0044] When there are display changes within the audio time range, the processor divides the audio time range into consecutive time periods based on each display change. The processor arranges the display change times falling within the voice time range from earliest to latest, taking the start time of the voice time range as the starting point of the first time segment and the first display change time as the boundary between the first time segment and the next time segment. When there are multiple display change times, the boundaries of the intermediate time segments are successively taken from two adjacent display change times, and the last time segment is formed from the last display change time to the end time of the voice time range. The time segments are arranged consecutively in chronological order, and each display change time corresponds to the boundary between adjacent time segments.

[0045] The processor associates each time period with its corresponding visual state. Based on the visual object records before and after each display change time in the visual state record, the processor determines the visual state corresponding to each time period; the time period before the first display change time is associated with the visual state before that display change time, the time period between two adjacent display change times is associated with the visual state after the previous display change time takes effect and before the next display change time takes effect, and the time period after the last display change time is associated with the visual state after that display change time takes effect; for display change times located at the boundary of adjacent time periods, the display change time is used as the starting point of the time period after the display change, so that each time period and visual state have a defined time boundary.

[0046] The processor reads the start and end times of each identification candidate that makes up the target description content; Based on the fact that each time period has been associated with the corresponding visual state, the processor reads the start and end times of each recognition candidate in the order of the speech positions in the target description. When multiple recognition candidates are retained for the same speech position, the start and end times of each recognition candidate are read respectively, and the established correspondence between recognition candidates, speech positions and start and end times is maintained.

[0047] When the start and end times of the candidate are in the same time period, the processor will associate the candidate with the visual state corresponding to the time period. The processor compares the start and end times of the candidate recognition with the start and end times of each time period. When the start and end times are both within the same time period, the corresponding candidate recognition is associated with the visual state corresponding to that time period. When the start time of the candidate recognition is equal to a certain display change time, the corresponding start position is assigned to the visual state after the display change. When the end time of the candidate recognition is equal to a certain display change time, the corresponding end position is assigned to the visual state before the display change, so that the candidate recognition has a definite visual state affiliation at the time period boundary.

[0048] When the start and end times of the identification candidate are located on both sides of the display change time, the processor will associate the identification candidate with the visual state before and after the display change to obtain the visual state corresponding to each identification candidate. When the start time of an identification candidate is earlier than a certain display change time and the end time is later than the corresponding display change time, the processor determines that the identification candidate spans the display change time and associates the identification candidate with the visual state before and after the display change, respectively. When an identification candidate spans multiple consecutive display change times, the processor associates the identification candidate with the visual state corresponding to each time period covered by its start and end times, according to the chronological order of the display change times. For example, if the start time of an identification candidate is 2.1 seconds and the end time is 3.0 seconds, and a display change occurs every 2.6 seconds in the visual state record, then the corresponding identification candidate is associated with the visual state before and after 2.6 seconds, thereby obtaining one or more visual states corresponding to each identification candidate.

[0049] IX. Determination of the Identification Results of Target Description Content like Figure 3 As shown, the processor reads the visual object in the corresponding visual state for each recognition candidate at the same speech position in the target description content; The processor sequentially reads the reserved recognition candidates according to the speech position in the target description, and reads the visual object in the corresponding visual state according to the established correspondence between each recognition candidate and the visual state. When there are multiple recognition candidates for the same speech position, each recognition candidate reads the visual object in the corresponding visual state. When a recognition candidate corresponds to two or more visual states before and after the display change, the visual objects in each corresponding visual state are read respectively, so that the visual objects participating in the subsequent matching are limited to the range of the visual state actually corresponding to the recognition candidate.

[0050] The processor compares the recognition content of each recognition candidate with the object name, object type and display attributes of the visual object in the corresponding visual state; For each recognition candidate, the processor reads the recorded object name, object type, and display attributes of each visual object in the corresponding visual state, and compares them according to the content category expressed by the recognition content. When the recognition content represents the name of a visual object, it compares the recognition content with the text content corresponding to the object name. When the recognition content represents the type of a visual object, it compares the recognition content with the text content corresponding to the object type. When the recognition content represents the display position, display size, display color, or visibility status of a visual object, it compares the recognition content with the corresponding display attributes. If the text content of an object name or object type is the same as the identified content, or if both correspond to the same meaning in the content corresponding to the same object name or object type stored in the multimedia visual output terminal, then a correspondence is determined. For display attributes, the processor determines the corresponding display description content based on the current value of the corresponding display attribute in the display execution data, and compares the identified content with the display description content. If both correspond to the same display attribute value, then a correspondence is determined between the identified content and the corresponding display attribute. For example, in the multimedia visual output terminal, the object name of a visual object is recorded as "weather window," the object type as "window," the display position as the right side of the display area, and the display color as blue. When the identified content of a candidate object is "weather window," it corresponds to the object name; when the identified content is "window," it corresponds to the object type; and when the identified content is "right side" or "blue," it corresponds to the display description content corresponding to the display position or display color, respectively. The processor only reads and compares the corresponding object content based on the actual content category expressed by the identified content.

[0051] For visual objects with corresponding content, the processor identifies the corresponding visual object as the matching object of the corresponding recognition candidate; When the recognition content of a candidate corresponds to the object name, object type, or display attribute of a certain visual object, the processor records the corresponding visual object as the matching object of the recognition candidate. When a recognition candidate corresponds to a visual object, a matching object is recorded. When a candidate corresponds to two or more visual objects, each matching object is recorded separately, thereby preserving the matching results between the recognition candidate and the visual object in the corresponding visual state.

[0052] The processor reads the matching objects corresponding to different speech positions according to the speech positions in the target description content; The processor gathers the matching objects corresponding to each recognition candidate according to the order of each speech position in the target description, and maintains the established correspondence between the recognition candidates and the matching objects. When there are multiple recognition candidates for the same speech position, the processor also retains the matching objects corresponding to each recognition candidate, so that the matching objects for different speech positions can continue to search for common objects according to the speech position.

[0053] The processor searches for the same visual object that can be found in the matching objects corresponding to different speech positions. Starting with a matching object corresponding to a speech position, the processor compares it sequentially with matching objects corresponding to other speech positions to find the same visual object that can be identified as a candidate in different speech positions. When matching objects in different speech positions have the same object identifier, the corresponding matching objects are identified as the same visual object. When different speech positions correspond to different visual states and the object identifier of the visual object changes due to the display change, the processor continuously reads the correspondence between the visual objects before and after the display change in the visual state record to confirm whether the matching objects in different visual states can be identified as the same visual object along the correspondence. In this way, the recognition content of multiple speech positions is used to jointly constrain the visual object.

[0054] The processor retains the corresponding recognition candidates based on the same visual object found; For each found visual object, the processor searches back for recognition candidates that match the corresponding visual object in different speech positions and retains the corresponding recognition candidates. If a recognition candidate in a certain speech position cannot correspond to any found visual object, the corresponding recognition candidate is excluded. When two or more visual objects that can correspond to recognition candidates in each speech position are found, the matching relationship between each visual object and the corresponding recognition candidate is maintained, so that multiple visual objects that meet the target description can continue to enter the process of determining the candidate target visual object.

[0055] The processor combines the retained recognition candidates according to the order of speech to obtain the recognition result of the target description content; The processor organizes the matched and retained recognition candidates according to the order of each speech position in the target description content. When only one recognition candidate is retained at the same speech position, the recognition content corresponding to the recognition candidate is recorded at the corresponding speech position. When more than two recognition candidates are retained at the same speech position, the recognition content corresponding to each recognition candidate is retained separately at that speech position, and the different recognition contents are not connected consecutively in the order of front and back. After organizing according to the original order of different speech positions, the recognition result of the target description content is obtained, so that the recognition result maintains the order of each speech position and the different recognition candidates currently retained at the same speech position.

[0056] The processor identifies the visual object corresponding to the recognition result as the candidate target visual object; When the processor reads the recognition results that form the target description content, it can identify the visual objects of each speech position and determine the corresponding visual objects as candidate target visual objects. When there are two or more visual objects that satisfy the corresponding relationship, each visual object is determined as a candidate target visual object, and the matching relationship between each candidate target visual object and the corresponding recognition candidate is retained for further determination of the target visual object.

[0057] 10. Determining the target visual object When there is only one candidate target visual object, the processor will identify the candidate target visual object as the target visual object. The processor reads the number of candidate target visual objects corresponding to the recognition results of the target description content; when there is only one candidate target visual object, it indicates that the recognition candidates retained at different speech positions have been collectively associated with a unique visual object. The processor directly determines the corresponding candidate target visual object as the target visual object and retains the visual state, object name, object type, display attributes and object identifier corresponding to the target visual object.

[0058] When there are two or more candidate target visual objects, the processor compares the object name, object type and display attributes of each candidate target visual object; When there are two or more candidate target visual objects, the processor reads the object name, object type and display attributes that have been recorded for each candidate target visual object one by one, and compares them item by item according to the same content; the object name is compared with the object name, the object type is compared with the object type, and each display attribute is compared with the corresponding display attribute, thereby obtaining the same value and different value of each candidate target visual object in object name, object type and display attribute.

[0059] The processor determines the object attributes with different values ​​among the candidate target visual objects based on the comparison results; The processor determines at least two candidate target visual objects with different values ​​from the comparison results of object name, object type, and display attributes, and identifies the corresponding items as object attributes with different values. Object attributes correspond to content with different values ​​in object name, object type, or specific display attributes. The same value does not participate in the differentiation of the current candidate target visual objects. For example, if two candidate target visual objects have the same object type but are displayed on the left and right sides respectively, the display position will be identified as an object attribute with different values.

[0060] The processor searches for recognition candidates corresponding to different values ​​from the recognition candidates retained at each speech position in the target description content; Based on the different values ​​that have been determined, the processor reads the currently retained recognition candidates at each speech position in the target description content and compares the recognition content of the recognition candidates with the corresponding different values ​​in turn; when the recognition content of a certain recognition candidate corresponds to a different value in the object name, object type or display attribute of the candidate target visual object, the corresponding recognition candidate is determined as the recognition candidate corresponding to that different value.

[0061] The processor identifies and excludes candidate visual objects whose attributes do not match the identified candidate objects. Based on the found recognition candidates, the processor checks the values ​​of each candidate target visual object on the corresponding object attributes one by one. When the object attribute value of the candidate target visual object corresponds to the recognition content of the recognition candidate, the corresponding candidate target visual object is retained. When the object attribute value does not correspond to the recognition content of the recognition candidate, the corresponding candidate target visual object is excluded. For example, if the recognition candidate retained in the target description corresponds to the right side, and the values ​​of the two candidate target visual objects at the display position are the left and right sides respectively, then the candidate target visual object at the display position on the left side is excluded, and the candidate target visual object at the display position on the right side is retained.

[0062] The processor continues to identify candidate target visual objects whose attributes do not match the candidate object attributes until a candidate target visual object is obtained. If two or more candidate visual objects remain after one elimination, the processor continues to compare the object name, object type, and display attributes of the remaining candidate visual objects to determine that there are still object attributes with different values ​​among the remaining candidate visual objects. Then, it continues to search for recognition candidates corresponding to the different values ​​from the recognition candidates retained at each speech position in the target description content. For each recognition candidate that can distinguish the remaining candidate visual objects, the processor continues to eliminate candidate visual objects whose object attributes do not match, until the number of candidate visual objects is reduced to one.

[0063] The processor identifies the candidate target visual objects as the target visual objects; When the number of candidate target visual objects is reduced to one, the processor determines the final retained candidate target visual object as the target visual object and maintains the established correspondence between the target visual object and the recognition result of the target description content. When the target visual object exists continuously in two or more visual states corresponding to the target description content through the visual object correspondence, the processor reads the visual object correspondence between the corresponding visual states in the order of the display change time, determines the latest visual state corresponding to the target description content as the visual state where the target visual object is located, and takes the visual object determined along the correspondence in that visual state as the target visual object. Thus, the determination of the current corresponding object is based on the target visual object and its visual state.

[0064] When there are still two or more candidate visual objects that cannot be distinguished based on the target description, the processor stops generating visual control instructions for the candidate visual objects. When the processor has read the recognition candidates retained at each speech position in the target description and there are still two or more remaining candidate target visual objects that cannot be distinguished based on the recognition content of the corresponding recognition candidates, it stops further determining the target visual objects and stops generating visual control instructions for the candidate target visual objects.

[0065] XI. Determining the Current Corresponding Object like Figure 4 As shown, the processor compares the visual state of the target visual object with the current visual state; The processor reads the visual state of the target visual object as determined above, and reads the current visual state of the multimedia visual output terminal when it is performing the current processing; it determines the order of the two visual states based on the display change time in the visual state record. If the time range corresponding to the visual state of the target visual object includes the current time, it determines that the visual state of the target visual object and the current visual state are the same visual state; if one or more display changes have occurred after the visual state of the target visual object ends, it determines that the visual state of the target visual object is earlier than the current visual state.

[0066] When the visual state of the target visual object is the same as the current visual state, the processor will identify the target visual object as the current corresponding object. When two visual states are the same, the target visual object is still in the current visual state, and there are no subsequent display changes that need to be crossed in the visual state record; the processor directly identifies the target visual object as the current corresponding object and retains the corresponding object identifier and display attributes for visual control.

[0067] When the visual state of the target visual object is earlier than the current visual state, the processor reads the correspondence between visual objects between adjacent visual states in the order of the display change time. The processor starts from the first display change time after the visual state of the target visual object, and reads the display change times in the visual state record in order from earliest to latest. For each display change time read, it also reads the two adjacent visual states before and after the display change time, as well as the correspondence between the visual objects that have been written into the visual state record, so that the target visual object corresponds continuously to the current visual state according to the actual display change order.

[0068] The processor starts with the target visual object and maps the visual objects that have been determined in the previous visual state to the result visual objects in the next visual state. When performing the correspondence between adjacent visual states for the first time, the visual object already determined in the previous visual state is the target visual object; the processor reads the correspondence relationship that has been established between the target visual object and the adjacent visual state, and reads the result visual object in the next visual state according to the correspondence relationship; when entering the next adjacent visual state, the result visual object obtained from the previous correspondence is taken as the visual object already determined in the previous visual state, and the correspondence relationship between it and the result visual object in the next visual state is read again, thereby continuously passing on the visual object along the visual state record.

[0069] The processor continues to read the correspondence between visual objects between adjacent visual states in the order of the display changes, until the current visual state is reached; Each time an object mapping from the previous visual state to the next visual state is completed, the processor determines whether the next visual state is the current visual state. If the current visual state has not yet been reached, it continues to read the adjacent visual states and visual object mapping relationships corresponding to the next display change time, and continues to execute the next mapping with the determined result visual object. The above process is carried out sequentially according to the display change time until the current visual state is reached.

[0070] When a visual object corresponds to two or more result visual objects, the processor reads the recognition content that has already matched the visual object from the target description content; When reading the correspondence between visual objects in adjacent visual states, if a visual object that has been identified in the previous visual state corresponds to more than two result visual objects in the next visual state, the processor retains each result visual object and reads the recognition content in the target description content that has formed a matching relationship with the visual object. The processor then uses the existing recognition content in the target description content to continue to distinguish multiple result visual objects.

[0071] The processor compares the identified content with the object name, object type, or display attribute of each resulting visual object; The processor reads the object name, object type, and display attributes of each result visual object one by one, and compares them according to the content corresponding to the recognition content. When the recognition content corresponds to the object name, the object name of each result visual object is compared; when the recognition content corresponds to the object type, the object type is compared; when the recognition content corresponds to the display attribute, the corresponding display attribute is compared. Thus, the comparison result between the recognition content and each result visual object is obtained.

[0072] The processor determines a result visual object based on the comparison result and continues to correspond to the next visual state; Based on the aforementioned comparison results, the processor retains the result visual objects whose object names, object types, or display attributes correspond to the recognized content, and excludes result visual objects that cannot form a correspondence with the recognized content; when only one result visual object is retained, the corresponding result visual object is determined as the corresponding result of the aforementioned visual object in the next visual state; if the current visual state is still earlier than the current visual state, the processor continues to read the visual object correspondence between the next adjacent visual states using the already determined result visual object.

[0073] Upon reaching the current visual state, the processor identifies the obtained visual object as the current corresponding object; When the processor continuously reads the visual object correspondence between adjacent visual states and reaches the current visual state, it stops reading and determines the last corresponding visual object in the current visual state as the current corresponding object; thus, the current corresponding object and the target visual object maintain a continuous correspondence formed along the actual display change sequence.

[0074] When there is no corresponding result visual object in any adjacent visual state, the processor stops generating visual control instructions for the target visual object; In the process of successively corresponding from the visual state where the target visual object is located to the current visual state, if there is no result visual object corresponding to the visual object that has been determined in the previous visual state between a certain adjacent visual state, the object correspondence process terminates at the corresponding visual state, and the processor stops forming visual control instructions for the target visual object.

[0075] When there are two or more resulting visual objects that cannot be distinguished based on the target description, the processor stops generating visual control instructions for the target visual objects. If a visual object corresponds to two or more resulting visual objects, and the processor has already compared the recognition content that matches the visual object in the target description with the object name, object type, or display attribute of each resulting visual object, and there are still two or more resulting visual objects, the processor will stop continuing to match from the corresponding visual state to the current visual state and stop forming visual control instructions for the target visual object.

[0076] 12. Formation of visual control commands and changes in the display state of the corresponding object The processor reads the object identifier and display attributes of the current corresponding object; The processor reads the object identifier and display attributes corresponding to the current corresponding object from the current visual state; the object identifier determines the visual object to which the display control operation is actually performed, and the display attributes record the display state of the current corresponding object before the execution of the visual control instruction, and serve as the basis for comparing the display attributes before and after the execution of the visual control instruction.

[0077] The processor determines the display control operation in the multimedia visual output terminal corresponding to the control action based on the control action; The processor reads the previously determined control action and, according to the established correspondence between the control action and the display control operations supported by the multimedia visual output terminal, reads the display control operation corresponding to the control action, so that the control action determined in the continuous voice is implemented as a display control operation that the multimedia visual output terminal can execute.

[0078] The processor combines the object identifier of the current corresponding object with the display control operation to obtain visual control instructions; The processor combines the determined display control operation with the object identifier of the corresponding object and records it in the same visual control instruction. The object identifier determines the visual object to which the display control operation is applied, and the display control operation determines the display change to be performed. This results in a visual control instruction with a clear target and execution content.

[0079] The processor sends visual control commands to the multimedia visual output terminal; The processor sends the generated visual control instructions to the multimedia visual output terminal; the multimedia visual output terminal locates the corresponding object based on the object identifier in the visual control instructions, reads the corresponding display control operation in the visual control instructions, and enters the execution process of the corresponding display control operation.

[0080] The multimedia visual output terminal changes the display state of the corresponding object according to visual control instructions; The multimedia visual output terminal executes corresponding display control operations for the current corresponding object identified in the visual control command, causing the current corresponding object to change its display state according to the control action; the visual object and display attributes generated by the execution of the display control operation are used as the display result after the execution of the visual control command, and the processor continues to read them to update the visual state record.

[0081] 13. Updating Visual Status Records After the visual control instruction is executed, the processor reads the visual object and display attributes after the visual control instruction is executed; When the multimedia visual output terminal completes the display control operation corresponding to the visual control instruction, the processor reads the visual object and corresponding display attributes formed by the actual effect of the display control operation; the visual object before the execution of the visual control instruction adopts the currently determined corresponding object, and the display attributes before the execution adopt the display attributes read before the formation of the visual control instruction, thereby obtaining the visual object and display attributes before and after the execution of the same visual control instruction.

[0082] The processor determines the time when the visual control instruction takes effect as the time of this display change; When the display control operation corresponding to the visual control command actually takes effect and causes the multimedia visual output terminal to form the display result after execution, the processor reads the corresponding effective time and records the effective time as the display change time. The display change time adopts the same timing benchmark as the aforementioned display change time, which represents the time position at which the visual control command causes the multimedia visual output terminal to change from the display state before execution to the display state after execution.

[0083] The processor compares the visual objects and display attributes before and after the execution of the visual control instructions; The processor reads and compares the object identifiers of the visual objects before and after the execution of the visual control instructions. At the same time, it reads the display attributes before and after the execution and compares the corresponding values ​​one by one. The comparison result of the object identifiers reflects whether the visual objects before and after the execution maintain the same object identifiers. The comparison result of the display attributes reflects whether the visual objects corresponding to the same object identifiers have changed their display attributes. When the object identifiers change, the processor further combines the display control operation corresponding to the visual control instructions to determine whether the current corresponding object has been replaced by a visual object with different object identifiers.

[0084] When the object identifiers are the same before and after the execution of the visual control instruction, and the display attributes change, the processor establishes a correspondence between the visual objects before and after the execution. When the object identifier comparison results are the same and the value of at least one display attribute changes before and after the execution of the visual control instruction, the processor establishes a correspondence between the visual object before execution and the visual object after execution based on the same object identifier, so that the correspondence between the same visual object before and after this display change is retained.

[0085] When a visual control instruction replaces the current corresponding object with a visual object that has a different object identifier, the processor establishes a correspondence between the current corresponding object and the replaced visual object based on the display control operation corresponding to the visual control instruction. When the current corresponding object before the execution of the visual control instruction has a different object identifier than the visual object after the execution, the processor reads the display control operation corresponding to the visual control instruction and determines whether the display control operation replaces the current corresponding object with the visual object after the execution. If a replacement relationship is confirmed, the current corresponding object before the execution of the visual control instruction is taken as the visual object before the display change, and the replaced visual object with a different object identifier is taken as the visual object after the display change. The processor then establishes a correspondence between the current corresponding object and the replaced visual object based on the display control operation.

[0086] The processor writes the time of this display change, the visual objects before and after the execution of the visual control instruction, the display attributes, and the corresponding relationships into the visual state record; After obtaining the display change time and the correspondence between visual objects before and after the execution of the visual control instruction, the processor writes the display change time into the corresponding time position according to the chronological order of the display change times already recorded in the visual state record. Simultaneously, it writes the visual objects, display attributes, and correspondences before and after the execution of the visual control instruction, so that the display change generated by the current visual control and the previously recorded display changes are continuously saved in chronological order.

[0087] The processor receives the updated visual state record; After writing the data corresponding to this display change, the visual state record is updated by adding the new visual state and visual object correspondence formed by the execution of the visual control command. The updated visual state record continues to retain the initial visual state, the time of each display change, the visual objects before and after the display change, and the correspondence between visual objects in adjacent visual states, and serves as the recording basis for determining the corresponding visual state and the current corresponding object in subsequent continuous speech processing.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimedia interactive visual control system based on intelligent voice recognition, characterized in that, It includes a voice acquisition module, a multimedia visual output terminal, and a processor, wherein the processor performs the following processing: The system collects continuous speech input from the user, performs speech recognition on the continuous speech, obtains multiple recognition candidates arranged in the order of speech, and records the start and end times of each recognition candidate in the continuous speech; it also records the time when the multimedia visual output terminal displays changes, the visual objects before and after the display changes, and the correspondence between the same visual object before and after the display changes, thus obtaining a visual state record. Based on the semantic relationship between each recognition candidate, the target description content describing the object to be controlled and the corresponding control action are determined from continuous speech, and the speech time range corresponding to the target description content is determined based on the start and end times of the recognition candidates that make up the target description content. The speech time range is compared with the display change time in the visual state record to determine one or more visual states corresponding to the speech time range, and the visual objects and display attributes in one or more visual states are read. When there are multiple recognition candidates at the same speech position, each recognition candidate is matched with the visual objects and display attributes in the corresponding visual state. The recognition result of the target description content is determined according to the recognition candidate that can form a corresponding relationship with the same visual object, and the visual object corresponding to the recognition result is determined as the target visual object. Determine the current corresponding object based on the relationship between the visual state of the target visual object and the current visual state; When the target visual object is in the current visual state, the target visual object is identified as the current corresponding object; When the visual state of the target visual object is different from the current visual state, the correspondence between the target visual object and the adjacent visual states is read sequentially according to the display change order in the visual state record to determine the current corresponding object of the target visual object in the current visual state. Based on the control action and the corresponding object, a visual control instruction is generated, and the multimedia visual output terminal is controlled to change the display state of the corresponding object according to the visual control instruction.

2. The system according to claim 1, characterized in that, The process of identifying candidates and determining start and end times includes: Speech recognition is performed on the audio data according to the acquisition order of continuous speech. Recognition results that correspond to the same speech position but have different recognition content are retained as recognition candidates, and each recognition candidate at the same speech position is associated with the corresponding audio data. Read the first and last audio frames from the audio data associated with each candidate for recognition, and determine the start and end times of the corresponding candidate for recognition based on the acquisition time of the first and last audio frames. The order of each speech position in continuous speech is determined by the acquisition time of the first audio frame in the audio data associated with each speech position, and the recognition candidates and start and end times corresponding to each speech position are recorded according to the order of arrangement.

3. The system according to claim 1, characterized in that, The process of forming the visual state record includes: Acquire display execution data sent to the multimedia visual output terminal, determine the display change operation that causes the addition, deletion, replacement or change of display attributes of visual objects from the display execution data, and read the visual object affected by the display change operation; Record the object identifier and display attributes of the visual object before the display change operation is executed, and read the object identifier and display attributes of the corresponding visual object after the display change operation is completed to obtain the visual object before and after the display change; The time when the display change operation takes effect is determined as the display change time, and the visual objects before and after the display change are recorded in the order of the display change time to obtain the visual state record.

4. The system according to claim 3, characterized in that, The process of determining the correspondence between the visual objects includes: Read the source visual object before the change operation is executed and the result visual object after the change operation is executed; When the source visual object and the result visual object have the same object identifier, the correspondence between the source visual object and the result visual object is established based on the same object identifier; when the display change operation replaces the source visual object with a result visual object with a different object identifier, or generates a result visual object with a different object identifier based on the source visual object, the correspondence between the source visual object and the result visual object is established based on the replacement relationship or generation relationship in the display execution data. For visual objects that are not affected by display change operations and exist continuously before and after the display change, establish a correspondence between the object before and after the display change based on the same object identifier, and write the formed correspondence and the corresponding display change time into the visual state record.

5. The system according to claim 2, characterized in that, The process of determining the target description content, control action, and speech time range includes: Read the display control operations supported by the multimedia visual output terminal, compare the recognition candidates of each voice position with the action content corresponding to the display control operation, determine the recognition candidates that can correspond to the display control operation as action candidates, and determine the display control operation corresponding to the action candidate as the control action; Based on the semantic relationship between action candidates and recognition candidates of other speech positions, recognition candidates describing the object of the controlled action are determined. The determined recognition candidates are combined according to the speech order to obtain the target description content, and different recognition candidates corresponding to the same speech position in the target description content are retained. Read the start and end times of each recognition candidate that makes up the target description content, determine the earliest start time as the start time of the target description content, and determine the latest end time as the end time of the target description content to obtain the speech time range corresponding to the target description content.

6. The system according to claim 5, characterized in that, The process of identifying the correspondence between candidates and visual states includes: The start and end times of the speech time range are compared with the display change times in the visual state recording to determine the display change times that fall within the speech time range. When there is no display change time within the speech time range, the visual state within the speech time range is determined as the corresponding visual state; when there is display change time within the speech time range, the speech time range is divided into continuous time periods by each display change time, and each time period is associated with the corresponding visual state. When the start and end times of the identification candidates that make up the target description content are in the same time period, the identification candidates are associated with the visual state corresponding to the time period; when the start and end times of the identification candidates are on both sides of the display change time, the identification candidates are associated with the visual states before and after the display change, respectively, to obtain the visual state corresponding to each identification candidate.

7. The system according to claim 6, characterized in that, The process of determining the identification result of the target description content includes: For each recognition candidate at the same speech position in the target description, the recognition content of each recognition candidate is compared with the object name, object type and display attributes of the visual object in the corresponding visual state. The visual object with corresponding content is determined as the matching object of the corresponding recognition candidate. Based on the speech location in the target description, search for the same visual object that can be identified as a candidate for each speech location from the matching objects corresponding to different speech locations; Based on the same visual object found, the corresponding recognition candidates are retained. The retained recognition candidates are combined according to the speech order to obtain the recognition result of the target description content, and the visual object corresponding to the recognition result is determined as the candidate target visual object.

8. The system according to claim 7, characterized in that, The process of determining the target visual object includes: When there is only one candidate target visual object, the candidate target visual object is determined as the target visual object; When there are two or more candidate target visual objects, compare the object name, object type and display attributes of each candidate target visual object to determine the object attributes with different values ​​among the candidate target visual objects. Search for recognition candidates corresponding to different values ​​from the recognition candidates retained at each speech position in the target description content, and exclude candidate target visual objects whose object attributes do not match the found recognition candidates until a candidate target visual object is obtained, and determine the obtained candidate target visual object as the target visual object. When there are still two or more candidate target visual objects that cannot be distinguished based on the target description, stop generating visual control instructions for the candidate target visual objects.

9. The system according to claim 8, characterized in that, The process of determining the current corresponding object includes: Compare the visual state of the target visual object with the current visual state. When the two are in the same visual state, the target visual object is identified as the current corresponding object. When the visual state of the target visual object is earlier than the current visual state, the correspondence between visual objects between adjacent visual states is read in the order of the display change time. Starting from the target visual object, the visual objects that have been determined in the previous visual state are mapped to the result visual objects in the next visual state, until the current visual state. When a visual object corresponds to two or more result visual objects, read the recognition content that has already matched the visual object in the target description content, and compare the recognition content with the object name, object type or display attribute of each result visual object. After determining a result visual object based on the comparison result, continue to correspond to the next visual state. When the current visual state is reached, the corresponding visual object is identified as the current corresponding object; when there is no corresponding result visual object in any adjacent visual state, or when there are two or more result visual objects that cannot be distinguished according to the target description, the generation of visual control instructions for the target visual object is stopped.

10. The system according to claim 9, characterized in that, The visual control and visual state recording update process includes: Read the object identifier and display attributes of the current corresponding object, and determine the display control operation in the multimedia visual output terminal corresponding to the control action based on the control action. Combine the object identifier and display control operation to obtain the visual control instruction. The visual control command is sent to the multimedia visual output terminal, the visual object and display attributes after the visual control command is executed are read, and the time when the visual control command takes effect is determined as the time of this display change. Compare the visual objects and display attributes before and after the execution of the visual control instruction. When the object identifier is the same and the display attribute changes, establish the correspondence between the visual objects before and after the execution. When the visual control instruction replaces the current corresponding object with a visual object with a different object identifier, establish the correspondence between the current corresponding object and the replaced visual object according to the display control operation corresponding to the visual control instruction. Write the time of the display change, the visual objects before and after the execution of the visual control command, the display attributes and their corresponding relationships into the visual state record to obtain the updated visual state record.

Images