Human body action and language instruction combined recognition system

Abstract

The invention discloses a human body action and language instruction combined recognition system, and particularly relates to the field of intelligent recognition, which comprises a language recognition module, an action recognition module, a mutual information value module, an independent analysis module, a fusion analysis module and an instruction generation module, the language recognition module is used for collecting human body language information and performing feature extraction on the human body language information to generate a language signal X; the action recognition module is used for collecting human body action information and performing feature extraction on the human body action information to generate an action signal Y; the mutual information value module constructs joint distribution from the language signal X and the action signal Y, and obtains probability distribution of the language signal X, probability distribution of the action signal Y and joint probability distribution of the language signal X and the action signal Y. According to the system, stable recognition can still be achieved in the complex environment with various accents and various visual angles, interaction errors caused by signal misjudgment are reduced, and a user can obtain natural and efficient experience in different interaction scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent identification, more particularly, to a human action and language instruction joint identification system. BACKGROUND

[0002] In the field of human-computer interaction, accurate recognition of human action and language instruction is the core basis for realizing natural interaction, and is widely used in intelligent home control, industrial human-computer cooperation, auxiliary driving and other scenes. However, the existing recognition technology still has many problems, and it is difficult to meet the interaction needs in complex environments:

[0003] First, the language instruction recognition is significantly affected by accent and ambiguity. Existing systems mostly rely on standard speech samples for training, and have poor adaptability to local accents, which can easily lead to misjudgment due to differences in tone and pronunciation. At the same time, the phenomenon of polysemy is common in natural language, and single language recognition lacks context verification, resulting in high ambiguity misrecognition rate.

[0004] Second, human action recognition is significantly limited by viewing angle and scene. Traditional action recognition relies on fixed-view cameras, and when the user's posture and shooting angle changes, it is easy to cause action misjudgment due to missing image features. Some systems introduce multi-view collection, but lack a consistency verification mechanism, and the action features at different angles are difficult to associate, resulting in insufficient action recognition accuracy in complex scenes. At the same time, the extraction of action features often ignores the spatiotemporal correlation of motion trajectories, and single-frame image analysis can easily lose dynamic information. Figure One Third, the joint recognition fusion logic is simple and inefficient. Existing technologies mostly rely on result splicing for action and language fusion, and do not establish a correlation at the feature level. When there is a delay or low correlation between action and language, forced fusion can introduce interference, resulting in insufficient improvement in joint recognition accuracy compared to single recognition. And lack of dynamic judgment mechanism, cannot choose independent analysis or fusion analysis strategy according to the correlation strength of the two, poor adaptability.

[0005] In addition, the instruction generation mechanism of existing systems is rigid: either relying on a single signal or forcibly fusing all signals, making it difficult to handle mixed scenarios such as clear language but ambiguous action, and clear action but ambiguous language, resulting in poor interaction flexibility and user experience. Therefore, there is an urgent need for a joint recognition system that can adapt to changes in accent and viewing angle, accurately determine signal correlation, and dynamically generate instructions.

[0006] In order to solve the above problems, a technical solution is provided.

[0007] SUMMARY SUMMARY

[0008] In order to overcome the above-mentioned defects of the prior art, embodiments of the present application provide a human action and language instruction joint identification system to solve the problems raised in the background art.

[0009] To achieve the above object, the present application provides the following technical solutions:

[0010] The human action and language instruction joint recognition system comprises a language recognition module, an action recognition module, a mutual information value module, an independent analysis module, a fusion analysis module and an instruction generation module.

[0011] The language recognition module is used for collecting human language information, and performing feature extraction on the human language information to generate a language signal X.

[0012] The action recognition module is used for collecting human action information, and performing feature extraction on the human action information to generate an action signal Y.

[0013] The mutual information value module constructs a joint distribution from the language signal X and the action signal Y, obtains a probability distribution of the language signal X, a probability distribution of the action signal Y and a joint probability distribution of the language signal X and the action signal Y, and calculates a mutual information value.

[0014] The independent analysis module separately analyzes the language signal X and the action signal Y when the language signal X and the action signal Y are independent of each other and have no correlation.

[0015] The fusion analysis module performs fusion analysis and processing when the language signal X and the action signal Y have correlation.

[0016] The instruction generation module automatically generates after the analysis is completed.

[0017] In a preferred embodiment, the language recognition module specifically comprises the following contents:

[0018] The standard speech sample is converted into variants of different accents, and the training data is expanded; meanwhile, the prosodic features of the original accent sample are retained, and the loss of accent recognition caused by excessive standardization is avoided.

[0019] During preprocessing, the speech signal collected by the array microphone is marked with accent features, and directional input is provided for subsequent feature extraction.

[0020] Before semantic feature extraction, the speech features that are not sensitive to accents are focused, the non-semantic features related to accents are weakened, and the speech of the target speaker is strengthened, so that purer input is provided for accent feature extraction.

[0021] When the semantic vector of the language feature extraction has multiple peaks, secondary judgment is triggered, the speech segment in the previous 3 seconds is combined, the user action feedback is combined to complete confirmation, and the language signal X is generated.

[0022] In a preferred embodiment, the action recognition module specifically comprises the following contents:

[0023] Model human motion, input human motion images to train the human motion model, and render images of a given perspective on each sample;

[0024] A multi-view diffusion model is used to generate a target view image with consistent content with the initial view image. A random view is first selected from the rendered image set as the reference view image. The camera transformation matrix and the reference view between the reference and target view images are then input into the multi-view diffusion model to obtain the target view image. The consistency measure between the views is determined by calculating the reconstruction loss between the generated target view image and the reference view.

[0025] Definition The rendered image set at the iteration is , the spatial representation of the rendered image set is , viewpoint and the camera parameter matrix Correlation, modeling the consistency between different views, using a multi-view diffusion model to transform from a reference view to a target view;

[0026]

[0027] in It is The spatial representation of the rendered image set at iteration In the selected reference view space, is the target view space, represents the time of the multi-view diffusion model, is the target viewpoint and reference viewpoints The transformation matrix of the camera matrix, is a multi-view diffusion model, is the target view space;

[0028] An action signal Y is generated based on the obtained result.

[0029] In a preferred embodiment, the mutual information value module operation specifically includes the following:

[0030] Using the time-frequency data of language signal X and action signal Y, we can obtain the probability distribution of language signal X. , the probability distribution of action signal Y , the joint probability distribution of language signal X and action signal Y , calculate the mutual information value of language signal X and action signal Y:

[0031] .

[0032] In a preferred embodiment, the independent analysis module runs specifically including the following:

[0033] When , it indicates that the language signal X and the action signal Y are independent of each other and have no correlation, at this time, the language signal X and the action signal Y are independently analyzed.

[0034] In a preferred embodiment, the fusion analysis module runs specifically including the following:

[0035] When , it indicates that the language signal X and the action signal Y have strong correlation between each other, at this time, the language signal X and the action signal Y are analyzed and processed by fusion analysis;

[0036] The language signal X and the action signal Y are decomposed in time and frequency dimensions, for the language signal X, the transform is defined as:

[0037] ;

[0038] Wherein, is the mother wave of the language signal X, is the scale parameter, is the time offset, is the decomposition coefficient of the language signal X;

[0039] For the action signal Y, the transform is defined as:

[0040] ;

[0041] Wherein, is the mother wave of the action signal Y, is the scale parameter, is the time offset, is the decomposition coefficient of the action signal Y.

[0042] In a preferred embodiment, the fusion correlation parameter is calculated, and the calculation formula is:

[0043] ;

[0044] Wherein, is the fusion correlation parameter.

[0045] In a preferred embodiment, the instruction generation module runs specifically including the following:

[0046] When the language signal X and the action signal Y are independently analyzed, the language instruction is generated first, and the action instruction is generated secondarily, for the personnel to select;

[0047] When the fusion analysis is performed on the language signal X and the action signal Y, the joint instruction is directly generated according to the obtained fusion correlation parameters.

[0048] The technical effects and advantages of the human action and language instruction joint recognition system of the present application are as follows:

[0049] 1. The language recognition module solves the problems of local accent adaptation difference, semantic ambiguity and noise interference through accent adaptation and semantic disambiguation design; the action recognition module breaks through the angle limitation and still maintains high action recognition stability in a complex scene by means of multi-view generation and consistency checking, so that the system can still stably recognize in a complex environment with various accents and changing angles, not only can adapt to the interaction requirements of different accents and random actions of family members in a home scene, but also can meet the recognition requirements of device noise interference and multi-angle operation of operators in an industrial scene, and has wide scene adaptation capability.

[0050] 2. The mutual information value module quantifies the correlation degree of signals to provide an objective basis for independent analysis or fusion analysis, and avoids interference caused by forced fusion; the fusion analysis module realizes feature-level fusion through time-frequency decomposition and parameter calculation, so that the joint recognition effect is significantly better than single recognition; the instruction generation module dynamically outputs instructions according to the correlation strength, generates joint instructions that fit the intention when the correlation is strong, and provides a selection space when the correlation is weak, so that this mechanism not only guarantees accurate interpretation of collaborative signals and reduces interaction errors caused by signal misjudgment, but also takes into account the interaction requirements of single signal dominant scenes, so that users can obtain natural and efficient experience in different interaction scenes. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The figure is a structural schematic diagram of the human action and language instruction joint recognition system of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0053] Embodiment 1

[0054] Figure 1 The human action and language instruction joint recognition system of the present application is given, which includes a language recognition module, an action recognition module, a mutual information value module, an independent analysis module, a fusion analysis module and an instruction generation module.

[0055] The language recognition module is configured to collect human language information and perform feature extraction on the human language information to generate a language signal X.

[0056] The action recognition module is configured to collect human action information and perform feature extraction on the human action information to generate an action signal Y.

[0057] The mutual information value module is configured to construct a joint distribution from the language signal X and the action signal Y, obtain a probability distribution of the language signal X, a probability distribution of the action signal Y, and a joint probability distribution of the language signal X and the action signal Y, and calculate a mutual information value.

[0058] The independent analysis module is configured to separately analyze the language signal X and the action signal Y when the language signal X and the action signal Y are independent of each other and have no correlation.

[0059] The fusion analysis module is configured to perform fusion analysis and processing when the language signal X and the action signal Y have correlation.

[0060] The instruction generation module is configured to automatically generate after the analysis is completed.

[0061] The language recognition module specifically includes the following contents:

[0062] The standard speech samples are converted into variants of different accents, and the training data is expanded; meanwhile, the prosodic features of the original accent samples are retained to avoid loss of accent recognition due to excessive standardization.

[0063] During preprocessing, the speech signal collected by the array microphone is marked with accent features to provide directional input for subsequent feature extraction.

[0064] Before semantic feature extraction, the speech features that are not sensitive to accents are focused on, the non-semantic features related to accents are weakened, and the target speaker's speech is strengthened to provide purer input for accent feature extraction.

[0065] When the semantic vector of the language feature extraction has multiple peaks, a secondary judgment is triggered, the speech segment within the previous 3 seconds is combined, and the user action feedback is combined to complete confirmation to generate the language signal X.

[0066] The training data is expanded by generating multi-accent variant samples, and the prosodic features are retained to adapt the model to various accents and avoid misjudgment due to accent differences, effectively improving the recognition accuracy under different accents; the accent features are marked in the preprocessing stage to make the subsequent extraction more targeted; the accent-insensitive features are focused on, the non-semantic features are weakened, the target speech is strengthened, the accent and noise interference is reduced, and the feature extraction accuracy is improved; when the semantic vector has multiple peaks, a secondary judgment is triggered, the speech segment within the previous 3 seconds is combined, and the user action feedback is combined to effectively reduce the semantic ambiguity misrecognition rate.

[0067] The action recognition module specifically includes the following contents:

[0068] Modeling human actions, inputting human action images to train a human action model, and rendering images of a given view angle on each sample;

[0069] Generating a target view image with consistent content with the initial view image using a multi-view diffusion model, first selecting a random view as a reference view image from the rendered image set, then inputting the camera transformation matrix between the reference view image and the target view image and the reference view into the multi-view diffusion model to obtain the target view image, and the consistency between the views is determined by calculating the reconstruction loss between the generated target view image and the reference view;

[0070] The rendered image set at the first iteration is defined as , the spatial representation of the rendered image set is , the view is associated with the camera parameter matrix , the consistency between different views is modeled, and the multi-view diffusion model is used for conversion from the reference view to the target view;

[0071]

[0072] wherein is the spatial representation of the rendered image set at the first iteration , the spatial of the selected reference view in the rendered image set is , the spatial of the target view is , the time of the multi-view diffusion model is represented as , the transformation matrix of the camera matrix of the target view and the reference view , the multi-view diffusion model is , the target view space is ; and According to the obtained results, an action signal Y is generated.

[0073] Modeling human actions, inputting human action images to train a human action model, and rendering images of a given view angle on each sample;

[0074]

[0075] ​​A reference view image is randomly selected from the set of rendered images, and a camera transformation matrix is combined with the reference view and the target view, including parameters such as view angle and position. The reference view is input into the multi-view diffusion model to generate a target view image consistent with the content of the reference view. By calculating the reconstruction loss of the generated target view image and the reference view, the lower the loss value, the higher the consistency, the action feature continuity between views is verified, and the invalid images with poor consistency are removed, and the valid views that meet the action logic are retained. The rendered image set of the nth iteration and its spatial representation are defined, the camera parameter matrix is associated with the view, and the multi-view diffusion model is used to complete the conversion from the reference view to the target view. During the conversion process, the accuracy of the view angle association is ensured through the camera transformation matrix. After multiple iterations, the spatial representation of the rendered image set is continuously optimized, and the action features of different views form a stable association. Finally, based on the optimized view features and the consistency verification result, the spatiotemporal features of the action, such as the start and end time of the action and the motion amplitude, are extracted to generate the action signal Y.

[0076] By modeling human actions and rendering images of a given view angle, and then generating target view images using a multi-view diffusion model, the model is exposed to a variety of view angle samples. This allows the model to accurately recognize actions even if the shooting angle changes in practical applications, reducing recognition errors caused by view angle differences and improving applicability in complex environments.

[0077] The reconstruction loss is used to measure the consistency of the views Figure One , and the camera transformation matrix and multi-view diffusion model are used to achieve accurate conversion from the reference view to the target view. This approach effectively ensures the stability of action features in different views and avoids inaccurate action feature extraction caused by view deviations, providing a reliable feature basis for subsequent action recognition.

[0078] Through multiple iterations to optimize the spatial representation of the rendered image set, consistent recognition results are obtained by combining the multi-view conversion logic to generate the action signal Y. This process fully considers the correlation and consistency of actions in different views, making the generated action signal more consistent with real action situations and providing strong support for subsequent operations based on the action signal, such as interaction.

[0079] The mutual information value module specifically includes the following content:

[0080] Using the time-frequency data of the language signal X and the action signal Y, the probability distribution of the language signal X , the probability distribution of the action signal Y , and the joint probability distribution of the language signal X and the action signal Y are obtained, and the mutual information value of the language signal X and the action signal Y is calculated:

[0081] .

[0082] By calculating the mutual information value of the language signal X and the action signal Y, the dependence of both in the time-frequency domain is quantified, providing a clear basis for subsequent independent analysis or fusion analysis, avoiding analysis deviation caused by subjective judgment of signal correlation. The mutual information value intuitively reflects the cooperativity of language and action, such as high mutual information corresponding to strong correlation and low mutual information corresponding to weak correlation, so that the system can select an analysis strategy based on objective data, improving the scientific nature of the overall interaction.

[0083] The independent analysis module runs and includes the following specific contents:

[0084] When , it indicates that the language signal X and the action signal Y are independent of each other and have no correlation, and the language signal X and the action signal Y are analyzed independently at this time.

[0085] When the language and action signals are independent of each other, independent analysis is used to avoid information interference caused by forced fusion, ensuring the analysis accuracy of a single signal, such as clear language instructions, focusing on language signal processing, and analyzing irrelevant signals respectively, which can fully exploit the effective information of a single signal and avoid ignoring the effective instructions of one party due to low correlation, ensuring that information is not wasted.

[0086] The fusion analysis module runs and includes the following specific contents:

[0087] When , it indicates that the language signal X and the action signal Y have strong correlation between each other, and the language signal X and the action signal Y are analyzed and processed by fusion at this time.

[0088] The language signal X and the action signal Y are decomposed in two dimensions of time and frequency. For the language signal X, the transformation is defined as:

[0089] ;

[0090] Wherein, is the mother wave of the language signal X, is the scale parameter, is the time offset, is the decomposition coefficient of the language signal X;

[0091] For the action signal Y, the transformation is defined as:

[0092] ;

[0093] Wherein, is the mother wave of the action signal Y, is the scale parameter, is the time offset, is the decomposition coefficient of the action signal Y.

[0094] The fusion correlation parameter is calculated as follows:

[0095]

[0096] wherein, is the fusion correlation parameter.

[0097] When the language and action are strongly associated, through time-frequency decomposition and fusion parameter calculation, the features of the two are accurately matched in the time and frequency dimensions, such as the hand-raising action and the opening instruction being synchronized in time and frequency feature association, extracting the collaborative semantics and improving the integrity of the instruction understanding. The fusion analysis can combine the advantages of the two signals, the language conveying clear semantics and the action supplementing scene information, when a single signal has noise or ambiguity, such as language with noise but clear action, the other signal can be used to assist correction, improving the stability of the overall analysis.

[0098] The instruction generation module specifically includes the following contents:

[0099] When the language signal X and the action signal Y are analyzed independently, the language instruction is generated first, and the action instruction is generated second, for the user to choose;

[0100] When the language signal X and the action signal Y are analyzed by fusion, the joint instruction is directly generated according to the obtained fusion correlation parameter.

[0101] In weak association, the language instruction is generated first and the action instruction is generated second, meeting the interaction demand of single signal dominance; in strong association, the joint instruction is directly generated, reflecting the collaborative intention of language and action, and outputting a more accurate joint instruction that meets the user's real needs. In weak association, the user is provided with a selection space, and in strong association, the precise joint instruction is directly outputted, taking into account the autonomous selection in complex scenarios and the efficient response in simple scenarios, optimizing the user interaction experience.

[0102] The fusion correlation parameter is a quantitative indicator of the association strength of the language signal X and the action signal Y, and the corresponding rules of the parameter value and the instruction collaboration need to be determined first. For example:

[0103] When the fusion correlation parameter is greater than 0.8, it is determined as strong collaboration, and the language core semantics and the action core features are directly extracted to form a joint instruction;

[0104] When the parameter is between 0.5 and 0.8, it is determined as weak collaboration, with language semantics as the main and action features supplementing the scene;

[0105] When the parameter is between 0.3 and 0.5, it is determined as potential collaboration, retaining the information of both and labeling the correlation degree.

[0106] The time-frequency decomposition coefficients of the language signal X and the corresponding parameters of the action signal Y are combined to ensure the consistency of the instruction in the time and frequency dimensions:​

[0107] The time offset parameter is used to determine the occurrence time sequence of language and action, such as language is sent at t=1.2s, action occurs at t=1.3s, and the time sequence difference is less than 0.5s, which is regarded as a synchronous instruction;

[0108] The scale parameter is used to match the feature frequency bands of the two, such as the language semantic features are concentrated in the 2kHz frequency band, and the action motion features respond to the corresponding low frequency band, and the frequency band overlap enhances the reliability of the instruction.

[0109] The interference information unrelated to the time sequence dislocation and the frequency band is removed to ensure the logical coherence of the joint instruction.

[0110] The above formulas are dimensionless to obtain numerical calculation, the formula is obtained by software simulation of a large amount of data to obtain the latest real situation, and the preset parameters and threshold values in the formula are set by a person skilled in the art according to the actual situation.

[0111] The above embodiments can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, the above embodiments can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0112] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing embodiments, which will not be repeated here.

[0114] In several embodiments provided in the present application, it should be understood that the disclosed system and device can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0115] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, which can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0116] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0117] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0118] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0119] Finally, the above merely provides the preferred embodiments of the present application, but is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A human motion and language instruction joint recognition system, characterized by: It includes language recognition module, action recognition module, mutual information value module, independent analysis module, fusion analysis module and instruction generation module; The language recognition module is used to collect human body language information, extract features from the human body language information, and generate language signal X; The motion recognition module is used to collect human motion information, extract features from the human motion information, and generate motion signal Y; The mutual information value module constructs a joint distribution from the language signal X and the action signal Y, obtains the probability distribution of the language signal X, the probability distribution of the action signal Y, and the joint probability distribution of the language signal X and the action signal Y, and calculates the mutual information value; The independent analysis module analyzes the language signal X and the action signal Y separately when the language signal X and the action signal Y are independent of each other and have no correlation; The fusion analysis module performs fusion analysis processing when the language signal X and the action signal Y are correlated; The instruction generation module is automatically generated after the analysis is completed.

2. The human body motion and language instruction combined recognition system according to claim 1, characterized in that: The operation of the language recognition module specifically includes the following: Convert standard speech samples into variants with different accents to expand training data; at the same time, retain the prosodic characteristics of the original accent samples to avoid loss of accent recognition due to over-standardization; During preprocessing, the speech signal collected by the array microphone is marked with accent features to provide directional input for subsequent feature extraction; Before semantic feature extraction, we focus on accent-insensitive speech features, weaken accent-related non-semantic features, and strengthen the target speaker's voice to provide purer input for accent feature extraction. When multiple peaks appear in the semantic vector extracted from the language features, a secondary judgment is triggered. The voice clips within the previous 3 seconds are combined with the user action feedback to complete the confirmation and generate the language signal X.

3. The human body motion and language instruction combined recognition system according to claim 2, characterized in that: The operation of the action recognition module specifically includes the following: Model human motion, input human motion images to train the human motion model, and render images of a given perspective on each sample; A multi-view diffusion model is used to generate a target view image with consistent content with the initial view image. A random view is first selected from the rendered image set as the reference view image. The camera transformation matrix and the reference view between the reference and target view images are then input into the multi-view diffusion model to obtain the target view image. The consistency measure between the views is determined by calculating the reconstruction loss between the generated target view image and the reference view. Definition The rendered image set at the iteration is , the spatial representation of the rendered image set is , viewpoint and the camera parameter matrix Correlation, modeling the consistency between different views, using a multi-view diffusion model to transform from a reference view to a target view; ； in It is The spatial representation of the rendered image set at iteration In the selected reference view space, is the target view space, represents the time of the multi-view diffusion model, is the target viewpoint and reference viewpoints The transformation matrix of the camera matrix, is a multi-view diffusion model, is the target view space; An action signal Y is generated based on the obtained result.

4. The human body motion and language instruction combined recognition system according to claim 3, characterized in that: The operation of the mutual information value module specifically includes the following: Using the time-frequency data of language signal X and action signal Y, we can obtain the probability distribution of language signal X. , the probability distribution of action signal Y , the joint probability distribution of language signal X and action signal Y , calculate the mutual information value of language signal X and action signal Y: 。 5. The human body motion and language instruction combined recognition system according to claim 4, characterized in that: The independent analysis module operation includes the following specific contents: exist , it indicates that the language signal X and the action signal Y are independent of each other and have no correlation. At this time, the language signal X and the action signal Y are analyzed independently.

6. The human body motion and language instruction combined recognition system according to claim 5, characterized in that: The operation of the fusion analysis module specifically includes the following: exist When , it indicates that the language signal X and the action signal Y are strongly correlated with each other, and the language signal X and the action signal Y are fused and analyzed; The language signal X and the action signal Y are decomposed in the time and frequency dimensions. For the language signal X, its transformation is defined as: ； in, It is the language signal X mother wave, is the scale parameter, is the time offset, is the decomposition coefficient of the language signal X; For the action signal Y, its transformation is defined as: ； in, is the action signal Y mother wave, is the scale parameter, is the time offset, is the decomposition coefficient of the motion signal Y.

7. The human body motion and language instruction combined recognition system according to claim 6, characterized in that: Calculate the fusion-related parameters using the following formula: ； in, are fusion-related parameters.

8. The human body motion and language instruction combined recognition system according to claim 7, characterized in that: The instruction generation module operation specifically includes the following: When language signal X and action signal Y are analyzed independently, language instructions are generated first and action instructions are generated secondarily for personnel to select; When the language signal X and the action signal Y are fused and analyzed, a joint instruction is directly generated based on the obtained fusion-related parameters.

Images