Multimodal fusion system based on audio-visual signs during movement

Abstract

The present invention relates to the field of data processing technology, and in particular to a multimodal fusion system based on audiovisual signs during movement, comprising: an audiovisual module, which is used to feed back three-dimensional scene images and audio to users; a binding module, which is used to receive audio fed back to users from the audiovisual module, segment the audio, and bind each group of segmented sub-audio to a three-dimensional model in the three-dimensional scene image fed back to users by the audiovisual module; and a configuration module, which is used to configure a three-dimensional scene space rectangular coordinate system, perceive user movement parameters in real time, and identify user position information in the three-dimensional scene. The present invention provides feedback direction fusion for audio feedback of users in virtual scenes by segmenting audio and binding sub-audio obtained by segmentation to virtual models in virtual scenes, so that when users walk in virtual scenes, the virtual scene audio can be fed back to users more realistically and vividly as the virtual scenes change, thereby improving the user's virtual scene experience.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multimodal fusion system based on audiovisual signs during movement. Background Art

[0002] VR virtual scene is a digital environment created using virtual reality technology, which enables users to experience it immersively. It is often used in education, games, tourism, architecture, medical care, industry, exhibitions and other fields.

[0003] The invention patent with application number 202110536436.2 discloses an AR / VR virtual reality fusion camera control method, including an RGB camera collector, a single-chip microcomputer, a VR processor, an AR processor, a sensor, and a display. The inner cavity of the single-chip microcomputer is provided with an image processing module and a radio frequency communication module, the inner cavity of the VR processor is provided with a marker recognition module, a storage module, an instruction transceiver element, a rendering module, and a coordinate generation module, the inner cavity of the AR processor is provided with a three-dimensional coordinate tracking module, a GPS module, a contour extraction module, a polygon conversion module, and a rendering module. The output end of the RGB camera collector is electrically connected to the input end of the single-chip microcomputer, the output end of the single-chip microcomputer is electrically connected to the input end of the VR processor, the output end of the VR processor is electrically connected to the input end of the AR processor, the output end of the AR processor is electrically connected to the input end of the sensor, and the output end of the sensor is electrically connected to the input end of the VR processor. The specific steps are as follows: In the first step, the RGB camera collector collects the image video; in the second step, the single-chip microcomputer processes the image, and its internal RF communication module transmits the image video to the VR processor; in the third step, the VR processor identifies the iconic substances inside the image video, and performs image segmentation and threshold segmentation on the image video, and then extracts the contour of the iconic substance and generates three-dimensional coordinates; in the fourth step, the affine invariance principle of the image video reconstructs the pose change matrix of the predefined marker coordinates to the current scene marker coordinates to complete the tracking and registration of the virtual information, and re-renders it and stores it in the storage module; in the fifth step, the AR processor tracks the three-dimensional coordinates through the GPS and retrieves the corresponding coordinate image from the storage module in the VR, and recognizes it again, extracts the contour, renders it, and then presents it on the display.

[0004] This application aims to solve the problem that "in existing control systems, VR and AR control systems exist independently and cannot be integrated into the control."

[0005] However, the existing VR technology has a poor integration effect on virtual scene changes and virtual scene sound effects, resulting in the sound effects changing stiffly with the virtual scene when the user is in the virtual scene, and failing to provide the user with a more realistic virtual scene experience.

[0006] To this end, we propose a multimodal fusion system based on audiovisual signs during movement. Summary of the invention

[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides a multimodal fusion system based on audiovisual signs during movement, which solves the technical problems raised in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] The multimodal fusion system based on audio-visual signs during movement includes:

[0010] The audio-visual module is used to feed back the three-dimensional scene image and audio to the user; the binding module is used to receive the audio fed back to the user in the audio-visual module, segment the audio, and bind each group of segmented sub-audio to the three-dimensional model in the three-dimensional scene image fed back to the user by the audio-visual module; the configuration module is used to configure the three-dimensional scene space rectangular coordinate system, perceive the user's movement parameters in real time, and identify the user's position information in the three-dimensional scene; the construction module is used to receive the user's position information in the three-dimensional scene identified in the configuration module, capture a group of three-dimensional models in the three-dimensional scene image closest to the user's position information in the three-dimensional scene, and use the user's position information in the three-dimensional scene and the captured The center position information of the three-dimensional model constructs a group of line segment models; the recognition module is used to receive the line segment model constructed in the construction module, synchronously construct an indication model, identify the similarity between the line segment model and the indication model, determine the indication model pointed to by the line segment model based on the similarity recognition result, and determine the audio-visual module through the indication model; the fusion module is used to receive the audio-visual module determined by the recognition module, and perform the fusion operation of the current user's three-dimensional scene image and audio with the audio-visual module.

[0011] Furthermore, the audio-visual module is integrated by sub-modules, including:

[0012] Wearable devices are used to store or receive three-dimensional scene images in real time and play them for users to read visually;

[0013] A wired headset is used to store and obtain the audio playback corresponding to the real-time received three-dimensional scene image for the user to read through hearing;

[0014] Among them, the left and right ear wearing positions of the wired earphones are each provided with eight groups of sound units, the eight groups of sound units are distributed in a circular array, the sound units are any one of dynamic sound units, moving iron sound units or electrostatic sound units, and the three-dimensional scene image played by the wearable device and the audio played by the wired earphones are adapted to each other and synchronously fed back to the user;

[0015] The eight groups of sound emitting units correspond to four groups of positive directions, up, down, left, and right, and a center direction between two adjacent groups of positive directions.

[0016] Furthermore, the binding module is internally provided with submodules, including:

[0017] The logic unit is used to set the audio segmentation logic when performing the segmentation operation in the binding module;

[0018] When the binding module receives the audio fed back to the user from the audio-visual module, the logic unit runs synchronously and feeds back the set audio segmentation logic to the binding module. The binding module performs segmentation processing on the audio through the audio segmentation logic to obtain several groups of sub-audio.

[0019] Furthermore, the audio segmentation logic set in the logic unit is:

[0020] Upload the audio to Adobe Audition, output the audio spectrum based on Adobe Audition, the horizontal axis of the output audio spectrum represents time, and the vertical axis represents sound volume / decibel, set a segmentation determination threshold, compare each audio frame in the audio spectrum with the segmentation determination threshold, use the position of the audio frame that meets the determination threshold as the segmentation position, segment the audio spectrum to obtain several groups of sub-audio spectra, identify the corresponding time period of each group of sub-audio spectra in the audio, and segment the audio based on the time period recognition result to obtain several groups of sub-audio;

[0021] The binding module performs an operation of binding the sub-audio with the three-dimensional model in the three-dimensional scene image, and the operation is manually performed by the system end user.

[0022] Furthermore, the configuration module is provided with submodules at a lower level, including:

[0023] The motion sign unit is used to provide the user with the conditions for moving in place and turning, and to sense the user's moving speed, direction, and distance in real time;

[0024] Among them, the motion sign unit runs synchronously with the wearable device and the wired earphones. When the motion sign unit runs initially, it feeds back a set of initial three-dimensional coordinates to the wearable device. The initial three-dimensional coordinates are fixed, and the initial three-dimensional coordinates are within the area defined by the three-dimensional scene space rectangular coordinate system based on the three-dimensional scene image.

[0025] Furthermore, the configuration module operates to receive the user's moving speed, moving direction, and moving distance sensed by the motion sign unit, and obtains the position information of the current user in the three-dimensional scene based on the user's moving speed, moving direction, moving distance, and the user's position information of the last positioning;

[0026] The configuration module first identifies the user's position information in the three-dimensional scene, and the user's position information of the last positioning applied is the initial three-dimensional coordinate fed back to the wearable device when the motion sign unit runs initially.

[0027] Furthermore, the indication model constructed in the recognition module is:

[0028] Obtain the distribution position information of each group of sound units, map the distribution position information of the sound units to the three-dimensional control, further identify the center position information of the sound units on the same side, and based on the sixteen groups of sound unit distribution position information and the two groups of center position information, connect the distribution position information of the sound units on the opposite side of one group of center position information with its relative center position information to obtain eight groups of sub-indicator models, and perform the same operation on the other group of center position information, and obtain a total of sixteen groups of sub-indicator models;

[0029] When identifying the similarity between the line segment model and the indicator model, a group of sub-indicator models in the indicator model are used to perform similarity identification with the line segment model;

[0030] The recognition module determines the operation of the audio-visual module by the indication model, that is, determines the operation of the sound unit in the audio-visual module by the indication model, and the sound unit determined by the recognition module is the sound unit corresponding to the sub-indication model with the highest similarity;

[0031] The identification module is provided with submodules at the lower level, including:

[0032] The storage module is used for storing the indication model.

[0033] Furthermore, the similarity calculation formula between the line segment model and the indication model is:

[0034] ;

[0035] Where: SIMM is the similarity between the line segment model and the sub-indicator model; k0 is the slope of the line segment model; k is the slope of the sub-indicator model; γ is the correction factor; L1 and L2 are the offset distances between the endpoints on the same side of the line segment model and the sub-indicator model;

[0036] Among them, the correction factor γ takes a value of 1 or -1. When k0>k, the correction factor γ takes a value of -1. When k0≤k, the correction factor γ takes a value of 1. After the indicator model is constructed, it is synchronously placed in the three-dimensional scene displayed by the three-dimensional scene image.

[0037] Furthermore, a refresh jump cycle is set in the fusion module. The fusion module continuously jumps to the configuration module operation stage based on the refresh jump cycle, controls the configuration module to refresh and operate, executes the user movement parameter perception, and identifies the user's position information in the three-dimensional scene. The refresh jump cycle is set according to the following rules: the faster the user moves and the more frequent the movement direction, the shorter the refresh jump cycle is set. Conversely, the longer the refresh jump cycle is set;

[0038] The fusion module uses the audio-visual module to perform the fusion operation of the current user's three-dimensional scene image and audio:

[0039] Obtain the sound unit determined in the recognition module, adjust the volume of the wired earphone, that is, adjust the volume of the determined sound unit, set the volume adjustment logic of the remaining sound units, and adjust the volume of the remaining sound units based on the adjustment logic;

[0040] The volume adjustment logic of the remaining sound-emitting units is expressed as:

[0041] ;

[0042] Where: m is the volume adjustment value of the remaining sound unit; m0 is the original volume value of the remaining sound unit; φ is a constant; d is the distance between the remaining sound unit and the sound unit determined by the recognition module;

[0043] Among them, d>φ>0, Table adjustment ratio.

[0044] Furthermore, the audio-visual module is interactively connected to a wearable device and a wired headset via a wireless network, the audio-visual module is interactively connected to a binding module via a wireless network, the binding module is interactively connected to a configuration module via a wireless network, the configuration module is interactively connected to a motion sign unit via a wireless network, the motion sign unit is interactively connected to the wearable device and the wired headset via a wireless network, the configuration module is interactively connected to a construction module, an identification module and a fusion module via a wireless network, and the identification module is interactively connected to a storage unit via a wireless network.

[0045] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects:

[0046] The present invention provides a multimodal fusion system based on audio-visual signs during movement. During operation, the system provides feedback direction fusion for audio feedback of users in virtual scenes by segmenting audio and binding sub-audios obtained by segmentation with virtual models in virtual scenes. When users walk in virtual scenes, the virtual scene audio can be fed back to users in a more realistic and vivid manner as the virtual scenes change, thereby improving the user's virtual scene experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a schematic diagram of the structure of a multimodal fusion system based on audio-visual signs during movement;

[0049] Figure 2 This is an example diagram of the indication model in the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The present invention will be further described below in conjunction with the embodiments.

[0052] Embodiment 1:

[0053] The multimodal fusion system based on audiovisual signs during movement in this embodiment is as follows: Figure 1 As shown, including:

[0054] Audio-visual module, used to provide users with feedback of 3D scene images and audio;

[0055] The audio-visual module is integrated by sub-modules, including:

[0056] Wearable devices are used to store or receive three-dimensional scene images in real time and play them for users to read visually;

[0057] A wired headset is used to store and obtain the audio playback corresponding to the real-time received three-dimensional scene image for the user to read through hearing;

[0058] Among them, the left and right ear wearing positions of the wired earphones are each provided with eight groups of sound units, which are distributed in a circular array. The sound units are any one of dynamic sound units, moving iron sound units or electrostatic sound units. The three-dimensional scene images played by the wearable device and the audio played by the wired earphones are adapted to each other and synchronously fed back to the user.

[0059] Eight groups of sound units correspond to four groups of positive directions, up, down, left, and right, and the center direction between two adjacent groups of positive directions;

[0060] A binding module, used to receive the audio fed back to the user by the audio-visual module, segment the audio, and bind each group of segmented sub-audio to the 3D model in the 3D scene image fed back to the user by the audio-visual module;

[0061] The binding module has submodules inside, including:

[0062] The logic unit is used to set the audio segmentation logic when performing the segmentation operation in the binding module;

[0063] In the stage where the binding module receives the audio fed back to the user in the audio-visual module, the logic unit runs synchronously and feeds back the set audio segmentation logic to the binding module. The binding module performs segmentation processing on the audio through the audio segmentation logic to obtain several groups of sub-audios;

[0064] Configuration module, used to configure the three-dimensional scene space rectangular coordinate system, real-time perception of user movement parameters, and identification of user position information in the three-dimensional scene;

[0065] The configuration module has submodules, including:

[0066] The motion sign unit is used to provide the user with the conditions for moving in place and turning, and to sense the user's moving speed, direction, and distance in real time;

[0067] The motion sign unit runs synchronously with the wearable device and the wired headset. When the motion sign unit runs initially, a set of initial three-dimensional coordinates is fed back to the wearable device, and the initial three-dimensional coordinates are fixed, and the initial three-dimensional coordinates are within the area defined by the three-dimensional scene space rectangular coordinate system based on the three-dimensional scene image;

[0068] A construction module is used to receive the position information of the user in the three-dimensional scene identified by the configuration module, capture a three-dimensional model in a group of three-dimensional scene images that is closest to the position information of the user in the three-dimensional scene, and construct a group of line segment models using the position information of the user in the three-dimensional scene and the center position information of the captured three-dimensional model;

[0069] The recognition module is used to receive the line segment model constructed in the construction module, synchronously construct the indication model, identify the similarity between the line segment model and the indication model, determine the indication model pointed to by the line segment model based on the similarity recognition result, and determine the audio-visual module through the indication model;

[0070] The identification module is provided with submodules at the lower level, including:

[0071] A storage module, used for storing the indication model;

[0072] The similarity calculation formula between the line segment model and the indicator model is:

[0073] ;

[0074] Where: SIMM is the similarity between the line segment model and the sub-indicator model; k0 is the slope of the line segment model; k is the slope of the sub-indicator model; γ is the correction factor; L1 and L2 are the offset distances between the endpoints on the same side of the line segment model and the sub-indicator model;

[0075] Among them, the correction factor γ takes a value of 1 or -1. When k0>k, the correction factor γ takes a value of -1. When k0≤k, the correction factor γ takes a value of 1. After the indicator model is constructed, it is synchronously placed in the three-dimensional scene displayed by the three-dimensional scene image.

[0076] A fusion module, used to receive the audio-visual module determined by the recognition module, and perform a fusion operation of the current user's three-dimensional scene image and audio with the audio-visual module;

[0077] A refresh jump cycle is set in the fusion module. The fusion module continuously jumps to the configuration module operation stage based on the refresh jump cycle, controls the configuration module to refresh and run, executes the user movement parameters, and identifies the user's position information in the three-dimensional scene. The setting of the refresh jump cycle is subject to the user's movement speed. The faster the user moves and the more frequent the movement direction, the shorter the refresh jump cycle is set. Conversely, the longer the refresh jump cycle is set;

[0078] In the fusion module, the fusion operation of the current user's three-dimensional scene image and audio is performed by the audio-visual module as follows:

[0079] Obtain the sound unit determined in the recognition module, adjust the volume of the wired earphone, that is, adjust the volume of the determined sound unit, set the volume adjustment logic of the remaining sound units, and adjust the volume of the remaining sound units based on the adjustment logic;

[0080] The volume adjustment logic of the remaining sound units is expressed as:

[0081] ;

[0082] Where: m is the volume adjustment value of the remaining sound unit; m0 is the original volume value of the remaining sound unit; φ is a constant; d is the distance between the remaining sound unit and the sound unit determined by the recognition module;

[0083] Among them, d>φ>0, Table adjustment ratio;

[0084] The audio-visual module is interactively connected to the wearable device and the wired headset through the wireless network, the audio-visual module is interactively connected to the binding module through the wireless network, the binding module is interactively connected to the configuration module through the wireless network, the configuration module is interactively connected to the motion sign unit through the wireless network, the motion sign unit is interactively connected to the wearable device and the wired headset through the wireless network, the configuration module is interactively connected to the construction module, the recognition module and the fusion module through the wireless network, and the recognition module is interactively connected to the storage unit through the wireless network.

[0085] In this embodiment, the audio-visual module runs to feed back three-dimensional scene images and audio to the user, the wearable device synchronously stores or receives the three-dimensional scene images in real time and plays them for the user to read visually, the wired earphones store and obtain the audio corresponding to the three-dimensional scene images in real time for the user to read auditorily, the binding module further receives the audio fed back to the user from the audio-visual module, segments the audio, and binds each group of segmented sub-audio to the three-dimensional model in the three-dimensional scene image fed back to the user by the audio-visual module, the logic unit synchronously sets the segmentation logic of the audio in the binding module when performing the segmentation operation, the configuration module post-operates to configure the three-dimensional scene space rectangular coordinate system, real-time perceives the user's movement parameters, and identifies the user's position information in the three-dimensional scene, the motion sign unit provides the user with real-time movement and turning conditions, and real-time perception The user's moving speed, moving direction, and moving distance are known, and then the construction module receives the position information of the user in the three-dimensional scene identified in the configuration module, captures a three-dimensional model in a group of three-dimensional scene images that are closest to the user's position information in the three-dimensional scene, and applies the user's position information in the three-dimensional scene and the central position information of the captured three-dimensional model to construct a group of line segment models. The recognition module runs to receive the line segment model constructed in the construction module, synchronously constructs an indication model, identifies the similarity between the line segment model and the indication model, and determines the indication model pointed to by the line segment model based on the similarity recognition result. The audio-visual module is determined by the indication model, and the storage module synchronously stores the indication model. Finally, the audio-visual module determined by the recognition module is received by the fusion module, and the audio-visual module is used to perform the fusion operation of the current user's three-dimensional scene image and audio;

[0086] Through the operation of the system in the above embodiment, a deeper integration optimization is made for the virtual scene experience of users in VR technology, ensuring that the virtual scene experience of the user is more realistic;

[0087] It should be noted that the wearable device is the VR glasses and the motion sign unit is the VR treadmill.

[0088] See also Figure 2 As shown, the figure is a three-dimensional figure drawn based on three-dimensional space. The combination of each group of line segments in the figure is used to represent a model, which is used as a reference for comparing the similarity of the line segment model and provides further operation data support for the operation of the above system.

[0089] Embodiment 2:

[0090] In terms of specific implementation, based on Example 1, this example refers to Figure 1 The multimodal fusion system based on audiovisual signs during movement in Example 1 is further described in detail:

[0091] The audio segmentation logic set in the logic unit is:

[0092] Upload the audio to Adobe Audition, output the audio spectrum based on Adobe Audition, the horizontal axis of the output audio spectrum represents time, and the vertical axis represents sound volume / decibel, set a segmentation determination threshold, compare each audio frame in the audio spectrum with the segmentation determination threshold, use the position of the audio frame that meets the determination threshold as the segmentation position, segment the audio spectrum to obtain several groups of sub-audio spectra, identify the corresponding time period of each group of sub-audio spectra in the audio, and segment the audio based on the time period recognition result to obtain several groups of sub-audio;

[0093] The binding module performs an operation of binding the sub-audio with the three-dimensional model in the three-dimensional scene image, and the system end user manually performs the operation;

[0094] The configuration module operates to receive the user's moving speed, moving direction, and moving distance sensed by the motion sign unit, and obtains the current user's position information in the three-dimensional scene based on the user's moving speed, moving direction, moving distance, and the user's position information of the last positioning;

[0095] The configuration module first identifies the user's position information in the three-dimensional scene, and the user's position information of the last positioning applied is the initial three-dimensional coordinate fed back to the wearable device when the motion sign unit runs the initial operation;

[0096] The indicator model constructed in the recognition module is:

[0097] Obtain the distribution position information of each group of sound units, map the distribution position information of the sound units to the three-dimensional control, further identify the center position information of the sound units on the same side, and based on the sixteen groups of sound unit distribution position information and the two groups of center position information, connect the distribution position information of the sound units on the opposite side of one group of center position information with its relative center position information to obtain eight groups of sub-indicator models, and perform the same operation on the other group of center position information, and obtain a total of sixteen groups of sub-indicator models;

[0098] When identifying the similarity between the line segment model and the indicator model, a group of sub-indicator models in the indicator model are used to perform similarity identification with the line segment model;

[0099] The recognition module determines the operation of the audio-visual module through the indication model, that is, the operation of the sound unit in the audio-visual module is determined through the indication model. The sound unit determined by the recognition module is the sound unit corresponding to the sub-indication model with the highest similarity.

[0100] In this embodiment, through the above settings, further operation data support is provided for the operation of the system in Example 1, and the construction logic of the indication model is further limited to ensure the stable construction of the indication model.

[0101] In summary, during the operation of the system in the above embodiment, through audio segmentation, the sub-audio obtained by segmentation is bound to the virtual model in the virtual scene, which provides a fusion of feedback direction for the audio feedback of the user in the virtual scene, so that when the user walks in the virtual scene, the virtual scene audio can be fed back to the user in a more real and vivid way as the virtual scene changes, thereby enhancing the user's virtual scene experience.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal fusion system based on audiovisual signs during movement, characterized by: include: Audio-visual module, used to provide users with feedback of 3D scene images and audio; A binding module, used to receive the audio fed back to the user by the audio-visual module, segment the audio, and bind each group of segmented sub-audio to the 3D model in the 3D scene image fed back to the user by the audio-visual module; Configuration module, used to configure the three-dimensional scene space rectangular coordinate system, real-time perception of user movement parameters, and identification of user position information in the three-dimensional scene; A construction module is used to receive the position information of the user in the three-dimensional scene identified by the configuration module, capture a three-dimensional model in a group of three-dimensional scene images that is closest to the position information of the user in the three-dimensional scene, and construct a group of line segment models using the position information of the user in the three-dimensional scene and the center position information of the captured three-dimensional model; The recognition module is used to receive the line segment model constructed in the construction module, synchronously construct the indication model, identify the similarity between the line segment model and the indication model, determine the indication model pointed to by the line segment model based on the similarity recognition result, and determine the audio-visual module through the indication model; A fusion module, used to receive the audio-visual module determined by the recognition module, and perform a fusion operation of the current user's three-dimensional scene image and audio with the audio-visual module; The audio-visual module is integrated by sub-modules, including: Wearable devices are used to store or receive three-dimensional scene images in real time and play them for users to read visually; A wired headset is used to store and obtain the audio playback corresponding to the real-time received three-dimensional scene image for the user to read through hearing; Among them, the left and right ear wearing positions of the wired earphones are each provided with eight groups of sound units, the eight groups of sound units are distributed in a circular array, the sound units are any one of dynamic sound units, moving iron sound units or electrostatic sound units, and the three-dimensional scene image played by the wearable device and the audio played by the wired earphones are adapted to each other and synchronously fed back to the user; The eight groups of sound-emitting units correspond to four groups of positive directions, up, down, left, and right, and the center direction between two adjacent groups of positive directions; The indication model constructed in the recognition module is: Obtain the distribution position information of each group of sound units, map the distribution position information of the sound units to the three-dimensional control, further identify the center position information of the sound units on the same side, and based on the sixteen groups of sound unit distribution position information and the two groups of center position information, connect the distribution position information of the sound units on the opposite side of one group of center position information with its relative center position information to obtain eight groups of sub-indicator models, and perform the same operation on the other group of center position information, and obtain a total of sixteen groups of sub-indicator models; When identifying the similarity between the line segment model and the indicator model, a group of sub-indicator models in the indicator model are used to perform similarity identification with the line segment model; The recognition module determines the operation of the audio-visual module by the indication model, that is, determines the operation of the sound unit in the audio-visual module by the indication model, and the sound unit determined by the recognition module is the sound unit corresponding to the sub-indication model with the highest similarity; The identification module is provided with submodules at the lower level, including: A storage module, used for storing the indication model; A refresh jump cycle is set in the fusion module. The fusion module continuously jumps to the operation stage of the configuration module based on the refresh jump cycle, controls the configuration module to refresh and run, executes the user movement parameter perception, and identifies the user's position information in the three-dimensional scene. The setting of the refresh jump cycle is subject to the following: the faster the user moves and the more frequent the movement direction, the shorter the refresh jump cycle is set. Conversely, the longer the refresh jump cycle is set; The fusion module uses the audio-visual module to perform the fusion operation of the current user's three-dimensional scene image and audio: Obtain the sound unit determined in the recognition module, adjust the volume of the wired earphone, that is, adjust the volume of the determined sound unit, set the volume adjustment logic of the remaining sound units, and adjust the volume of the remaining sound units based on the adjustment logic; The volume adjustment logic of the remaining sound-emitting units is expressed as: ； Where: m is the volume adjustment value of the remaining sound unit; m0 is the original volume value of the remaining sound unit; φ is a constant; d is the distance between the remaining sound unit and the sound unit determined by the recognition module; Among them, d>φ>0, d represents the adjustment ratio; The similarity calculation formula between the line segment model and the indication model is: ； Where: SIMM is the similarity between the line segment model and the sub-indicator model; k0 is the slope of the line segment model; k is the slope of the sub-indicator model; γ is the correction factor; L1 and L2 are the offset distances between the endpoints on the same side of the line segment model and the sub-indicator model; Among them, the correction factor γ takes a value of 1 or -1. When k0>k, the correction factor γ takes a value of -1. When k0≤k, the correction factor γ takes a value of 1. After the indicator model is constructed, it is synchronously placed in the three-dimensional scene displayed by the three-dimensional scene image.

2. The multimodal fusion system based on audiovisual signs during movement according to claim 1 is characterized in that: The binding module is internally provided with submodules, including: The logic unit is used to set the audio segmentation logic when performing the segmentation operation in the binding module; When the binding module receives the audio fed back to the user from the audio-visual module, the logic unit runs synchronously and feeds back the set audio segmentation logic to the binding module. The binding module performs segmentation processing on the audio through the audio segmentation logic to obtain several groups of sub-audio.

3. The multimodal fusion system based on audiovisual signs during movement according to claim 2 is characterized in that: The audio segmentation logic set in the logic unit is: Upload the audio to Adobe Audition, output the audio spectrum based on Adobe Audition, the horizontal axis of the output audio spectrum represents time, and the vertical axis represents sound volume / decibel, set a segmentation determination threshold, compare each audio frame in the audio spectrum with the segmentation determination threshold, use the position of the audio frame that meets the determination threshold as the segmentation position, segment the audio spectrum to obtain several groups of sub-audio spectra, identify the corresponding time period of each group of sub-audio spectra in the audio, and segment the audio based on the time period recognition result to obtain several groups of sub-audio; The binding module performs an operation of binding the sub-audio with the three-dimensional model in the three-dimensional scene image, and the operation is manually performed by the system end user.

4. The multimodal fusion system based on audiovisual signs during movement according to claim 1, characterized in that: The configuration module is provided with submodules at the lower level, including: The motion sign unit is used to provide the user with the conditions for moving in place and turning, and to sense the user's moving speed, direction, and distance in real time; Among them, the motion sign unit runs synchronously with the wearable device and the wired earphones. When the motion sign unit runs initially, it feeds back a set of initial three-dimensional coordinates to the wearable device. The initial three-dimensional coordinates are fixed, and the initial three-dimensional coordinates are within the area defined by the three-dimensional scene space rectangular coordinate system based on the three-dimensional scene image.

5. The multimodal fusion system based on audiovisual signs during movement according to claim 1 is characterized in that: The configuration module is operated to receive the user's moving speed, moving direction, and moving distance sensed by the motion sign unit, and obtain the current user's position information in the three-dimensional scene based on the user's moving speed, moving direction, moving distance, and the user's position information of the last positioning; The configuration module first identifies the user's position information in the three-dimensional scene, and the user's position information of the last positioning applied is the initial three-dimensional coordinate fed back to the wearable device when the motion sign unit runs initially.

6. The multimodal fusion system based on audiovisual signs during movement according to claim 1, characterized in that: The audio-visual module is interactively connected to a wearable device and a wired headset via a wireless network, the audio-visual module is interactively connected to a binding module via a wireless network, the binding module is interactively connected to a configuration module via a wireless network, the configuration module is interactively connected to a motion sign unit via a wireless network, the motion sign unit is interactively connected to the wearable device and the wired headset via a wireless network, the configuration module is interactively connected to a construction module, an identification module and a fusion module via a wireless network, and the identification module is interactively connected to a storage unit via a wireless network.

Images