Visual enhancement method and system and electronic equipment

Abstract

The invention discloses a visual enhancement method and system and electronic equipment, and relates to the field of AR, and the method comprises the steps: obtaining collected audio data; analyzing the audio data to obtain a sound source category, a sound source relative speed, a sound source relative motion direction and a sound source relative displacement change of a sound source corresponding to the audio data; determining a corresponding sound source image according to the sound source category; determining the color and visual change mode of the sound source image according to the sound source category, the sound source relative speed, the sound source relative motion direction and the sound source relative displacement change; and sending the sound source image, the color of the sound source image and the visual change mode to a display device of augmented reality AR glasses, so that the sound source image with the color and the visual change mode is displayed on the display device of the AR glasses. The corresponding sound source image, the sound source category, the sound source relative speed, the sound source relative movement direction, the sound source relative displacement change color and the visual change mode are determined through the sound source category, and visual display of the sound source is achieved to remind a user.

Description

Technical Field

[0001] The present application relates to the field of AR technology, and in particular to a vision enhancement method, system and electronic device. Background Art

[0002] AR (Augmented Reality) glasses are a type of smart wearable device. When a user wears AR glasses, the camera on the glasses can capture image signals of the user's surroundings and superimpose virtual images on a transparent display or micro-projector. Therefore, AR glasses can increase the user's perception of the real world.

[0003] However, there may be sounds from various sources in the user's surroundings, such as vehicle horns, bicycle bells, etc. For users, if they do not pay attention to the various sounds in the environment, there will be safety hazards, especially for the hearing impaired, where sound is even more important. Therefore, how to convert sound into visual on AR glasses to remind users is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, the present application provides a visual enhancement method, system and electronic device to achieve the purpose of converting sound into vision and reminding users. The specific scheme is as follows:

[0005] A first aspect of the present application provides a visual enhancement method, the visual enhancement method comprising:

[0006] Get the collected audio data;

[0007] Analyze the audio data to obtain a sound source category, a relative speed of the sound source, a relative movement direction of the sound source, and a relative displacement change of the sound source corresponding to the audio data;

[0008] Determine a corresponding sound source image according to the sound source category;

[0009] Determine the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source and the relative displacement change of the sound source;

[0010] The sound source image, the color of the sound source image and the visual change mode are sent to a display device of augmented reality AR glasses, so that the sound source image with the color and the visual change mode is displayed on the display device of the AR glasses.

[0011] In a possible implementation, the audio data includes: an audio spectrum signal, a sound source arrival angle signal, an audio amplitude difference, a time difference and a time-frequency feature, wherein the time-frequency feature is: a feature extracted by short-time Fourier transform analysis after framing the audio spectrum signal,

[0012] Analyzing the audio data to obtain the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source corresponding to the audio data, including:

[0013] Convert the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature into two-dimensional matrix forms respectively, wherein the rows of the two-dimensional matrix are set as data category dimensions and the columns of the matrix are set as time dimensions;

[0014] Input the audio spectrum signal in the form of a two-dimensional matrix, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature into a convolutional neural network (CNN) model, and obtain the first sound source relative speed, the first sound source relative motion direction and the first sound source relative displacement change output by the CNN model;

[0015] Inputting the audio spectrum signal in a two-dimensional matrix form, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature into a Doppler effect model, and obtaining the second sound source relative speed, the second sound source relative movement direction and the second sound source relative displacement change output by the Doppler effect model;

[0016] The first sound source relative speed, the first sound source relative movement direction and the first sound source relative displacement change are weightedly calculated with the second sound source relative speed, the second sound source relative movement direction and the second sound source relative displacement change to obtain the sound source relative speed, the sound source relative movement direction and the sound source relative displacement change.

[0017] In a possible implementation, the audio data includes: an audio spectrum signal,

[0018] Analyzing the audio data to obtain a sound source category of a sound source corresponding to the audio data includes:

[0019] The audio spectrum signal is framed, and the framed audio spectrum signal is converted into a two-dimensional matrix using Mel-frequency cepstrum coefficients, wherein the rows of the two-dimensional matrix are set as frequency dimensions, and the columns of the two-dimensional matrix are set as time dimensions;

[0020] The audio spectrum signal in the form of a two-dimensional matrix is ​​input into a convolutional neural network (CNN) model to obtain a sound source classification label ID output by the CNN model, and the sound source classification label ID is used as the sound source category.

[0021] In a possible implementation, the sound source category is a sound source classification label ID, and determining a corresponding sound source image according to the sound source category includes:

[0022] The virtual shape of the object corresponding to the sound source classification tag ID is used as the sound source image corresponding to the sound source category, and the virtual shape of the object is a pedestrian, an animal, or a vehicle.

[0023] In a possible implementation, the color indicates the degree of danger of the sound source to the user, and the visual change mode indicates the reminder mode of the sound source on the display device of the AR glasses.

[0024] The determining of the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source and the relative displacement change of the sound source comprises:

[0025] Determining the danger level of the sound source according to the sound source category;

[0026] Determining the relative movement of the sound source with respect to the user according to the relative speed of the sound source and the relative movement direction of the sound source;

[0027] The color and the visual change mode of the sound source image are determined according to the danger level of the sound source, the relative movement of the sound source and the relative displacement change of the sound source.

[0028] In a possible implementation, the danger level of the sound source includes low danger, medium danger and high danger.

[0029] Determining the color of the sound source image according to the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source includes:

[0030] When the sound source does not move relative to the sound source or the relative displacement of the sound source remains unchanged, the color is determined as follows: the low risk is a safety color, and the medium risk and the high risk are both the first degree of warning colors;

[0031] When the sound source moves relative to the sound source and the relative displacement of the sound source gradually increases, the color is determined in the following manner: the low risk is a safety color, the medium risk is a warning color of the first degree, and the high risk is a warning color of the second degree;

[0032] When the sound source moves relative to the user and the relative displacement of the sound source gradually decreases, the color is determined as follows: according to the relative speed of the sound source and the current distance between the sound source and the user, the low risk is one of the first degree of warning color, the second degree of warning color, and the third degree of warning color; the medium risk is one of the second degree of warning color and the third degree of warning color; and the high risk is one of the second degree of warning color and the third degree of warning color;

[0033] The danger level represented by the first level of warning color is lower than the danger level represented by the second level of warning color, and the danger level represented by the second level of warning color is lower than the danger level represented by the third level of warning color.

[0034] In a possible implementation, the danger level of the sound source includes low danger, medium danger and high danger.

[0035] Determining the visual change mode of the sound source image according to the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source includes:

[0036] When the sound source does not move relative to the sound source or the relative displacement of the sound source remains unchanged, the visual change mode is: displaying the sound source image at the display position;

[0037] When the sound source moves relatively and the relative displacement of the sound source gradually increases, the visual change mode is: the sound source image is displayed at the display position and the sound source image is gradually reduced;

[0038] When the sound source undergoes relative motion and the relative displacement of the sound source gradually decreases, the visual change mode is: the low-risk sound source image is displayed at the display position and the sound source image gradually increases, the medium-risk and high-risk sound source images are displayed at the display position and the sound source image gradually increases, and according to the relative speed of the sound source and the current distance between the sound source and the user, it is determined whether to increase the flashing display effect.

[0039] In a possible implementation, the visual enhancement method further includes:

[0040] Obtain a video containing the user's sign language movements, mouth shapes, and facial expressions;

[0041] Splitting the video into continuous image data,

[0042] Analyze the sign language action in the continuous image data to obtain a first predicted text corresponding to the sign language action;

[0043] Analyze the lip shape and facial expression in the continuous image data to obtain a second predicted text corresponding to the lip shape and the facial expression;

[0044] The target predicted text is sent to a display device of the AR glasses, so that the target predicted text is displayed on the display device of the AR glasses, and the target predicted text is one of the first predicted text and the second predicted text.

[0045] A second aspect of the present application provides a visual enhancement system, the visual enhancement system comprising:

[0046] An acquisition unit, used for acquiring collected audio data;

[0047] An analysis unit, configured to analyze the audio data to obtain a sound source category, a relative speed of the sound source, a relative movement direction of the sound source, and a relative displacement change of the sound source corresponding to the audio data;

[0048] An image recognition unit, used to determine a corresponding sound source image according to the sound source category;

[0049] A display determination unit, configured to determine the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source;

[0050] A sending unit is used to send the sound source image, the color of the sound source image and the visual change mode to the display device of the augmented reality AR glasses, so that the sound source image with the color and the visual change mode is displayed on the display device of the AR glasses.

[0051] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0052] The memory is used to store computer programs;

[0053] The processor is used to execute the computer program so that the electronic device can implement the visual enhancement method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0054] By means of the above technical scheme, the present application provides a visual enhancement method, system and electronic device. The method obtains the sound source category, relative speed of the sound source, direction of movement of the sound source and relative displacement change of the sound source corresponding to the audio data by analyzing the collected audio data, thereby determining the image display mode of the sound source on the display device of AR glasses according to the sound source category, relative speed of the sound source, relative movement direction of the sound source and relative displacement change of the sound source. The sound source category can determine the sound source image corresponding to the sound source, and the sound source category, relative speed of the sound source, relative movement direction of the sound source and relative displacement change of the sound source can determine the color and visual change mode of the sound source image. After determining the image display mode of the sound source on the display device of AR glasses, it can be sent to the display device of AR glasses, so that the sound source image with color and visual change mode is displayed on the display device of AR glasses. The method converts the analysis results of the audio data into the image display mode of the sound source on the display device of AR glasses, realizes the visual display of the sound source in the environment to remind the user, and the sound source also has corresponding color and visual changes on the display device of AR glasses, which can better remind the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0056] Figure 1 A flowchart of a visual enhancement method provided in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of a display sound source of a display screen provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of the structure of a visual enhancement system provided in an embodiment of the present application;

[0059] Figure 4 A hardware structure block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0061] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0062] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0063] The embodiment of the present application provides a visual enhancement method. The visual enhancement method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0064] The AR glasses involved in the visual enhancement method in this embodiment may include components such as a microphone, a camera, a display screen, a Bluetooth module, related connection circuits and a power supply. Input modules such as microphones and cameras can collect information about the surrounding environment. The Bluetooth module transmits the collected information to the data processing part for data processing, analysis and conversion. In this embodiment, the data processing part can be an application in the user's mobile terminal. Of course, under the condition of not affecting the weight and operation ability of the AR glasses as much as possible, the data processing part can also be a data processing module possessed by the AR glasses themselves. The Bluetooth module then transmits the data processing results of the data processing part to the display screen, and the display screen projects the information of the results. Therefore, the hardware part of the AR glasses in this embodiment only includes necessary I / O modules, communication modules and power modules, etc., and important data storage, data processing, data analysis and other processes are handed over to the application in the mobile terminal. Under the premise of ensuring the application performance of the AR glasses, the hardware weight and working heat level of the AR glasses can be reduced, and the wearing effect of the AR glasses can be effectively improved.

[0065] Reference Figure 1 , Figure 1 A flow chart of a visual enhancement method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a visual enhancement method provided in an embodiment of the present application may include steps S10 to S14, and these steps are described in detail below.

[0066] S10: Acquire collected audio data.

[0067] Among them, the audio data can be collected and obtained through the input component of the AR glasses. Specifically, the audio data can be collected and obtained through the microphone on the AR glasses. Multiple groups of microphones can be distributed on the AR glasses, and the multiple groups of microphones are respectively arranged at different positions of the AR glasses to form a microphone array, which is convenient for collecting audio, volume and other data of the surrounding environment. Therefore, the audio data in this embodiment mainly includes: sound data in the surrounding environment, such as the sound of vehicles, animal sounds, and the sound of pedestrians passing by. The audio data in this embodiment may include: audio spectrum signals, sound source arrival angle signals, audio amplitude differences, time differences and time-frequency characteristics.

[0068] Specifically, the audio spectrum signal can be a representation signal in the frequency domain obtained after the audio signal is transformed by Fourier transform or other similar transformations. It is a spectrum signal of the audio signal, which can reflect the intensity or amplitude of each frequency component in the audio signal. The sound source arrival angle signal (DOA, Direction of Arrival) can be: when the sound wave propagates from the sound source to the receiver array (microphone array), the angle corresponding to the signal between the direction of arrival of the sound wave and a reference direction of the receiver array (such as the normal direction of the array). In this embodiment, the sound source arrival angle signal specifically refers to: the angle corresponding to the signal between the direction of arrival of the sound wave of the sound source and a reference direction of the microphone array. The audio amplitude difference (ILD, Interaural Level Difference) can be: the intensity difference of the same source audio information received by microphones at different positions. The time difference (TDOA, Time Difference of Arrival) can be: the time difference of the same source audio information received by microphones at different positions. The time-frequency feature can be: the feature extracted by short-time Fourier transform analysis after the audio spectrum signal is framed.

[0069] This embodiment collects audio data in the surrounding environment through input devices such as microphones, and transmits the collected audio data to an application program at the user end through Bluetooth for data processing.

[0070] S11, analyzing the audio data to obtain the sound source category, relative speed, relative movement direction and relative displacement change of the sound source corresponding to the audio data;

[0071] S12. Determine a corresponding sound source image according to the sound source category.

[0072] The sound source categories in this embodiment may include pedestrians, animals, and vehicles, and the aforementioned categories may be further subdivided. For example, vehicles may include bicycles, electric bicycles, and family cars.

[0073] The collected audio data may include audio data of multiple sound sources, and for each sound source, this embodiment may determine the sound source category of the sound source by analyzing the audio spectrum signal of the sound source. The specific process may be as follows:

[0074] The audio spectrum signal is framed, and the Mel-frequency cepstrum coefficients are used to convert the framed audio spectrum signal into a two-dimensional matrix, the rows of the two-dimensional matrix are set as the frequency dimension, and the columns of the two-dimensional matrix are set as the time dimension; the audio spectrum signal in the form of a two-dimensional matrix is ​​input into a convolutional neural network (CNN) model to obtain the sound source classification label ID output by the convolutional neural network (CNN) model, and the sound source classification label ID is used as the sound source category.

[0075] Among them, after obtaining a long audio spectrum signal of a sound source collected by a microphone, the long audio spectrum signal can be framed by methods such as cyclic sampling method and indexing method to reduce the computational complexity. Mel frequency cepstral coefficient (MFCC) is an important feature in audio signal processing, which is used to capture the spectral characteristics of audio signals. In this embodiment, the feature matrix (two-dimensional matrix) of the audio spectrum signal can be obtained by Mel frequency cepstral coefficient.

[0076] The CNN (Convolutional Neural Networks) model can be a type of neural network that includes convolution calculations and has a deep structure. In this embodiment, before using the CNN model to perform the classification task of the two-dimensional matrix of the audio spectrum signal, the UrbanSound8K dataset is used to train the CNN model to optimize the feature extraction of the CNN model. The UrbanSound8K dataset is currently a widely used dataset for environmental sound classification research. The UrbanSound8K dataset can cover multiple types of sounds, such as cars, engines, traffic lights, bicycles, children playing, dogs barking, etc. In this embodiment, the feature matrix of the audio spectrum signal is input into the trained CNN model for sound source classification. In the CNN model, the feature extraction layer (which may include a convolution layer and a pooling layer) extracts the features of the feature matrix, and the fully connected layer maps the extracted features to the audio classification label, and the output layer outputs the sound source classification label ID corresponding to the audio classification label. The correspondence between the audio classification label and the sound source classification label ID can be shown in Table 1:

[0077] Table 1

[0078]

[0079] In this embodiment, the CNN model outputs the sound source classification label ID as the sound source category. Therefore, when determining the sound source category and the corresponding sound source image, the virtual shape of the object corresponding to the sound source classification label ID can be used as the sound source image corresponding to the sound source category. Among them, the virtual shape of the object can be: pedestrians, animals, or vehicles. For example, if the sound source classification label ID is B001, the corresponding audio classification label is vehicle (bicycle), then the virtual shape of the bicycle can be used as the sound source image corresponding to B001.

[0080] The relative speed of the sound source may refer to: the speed of the sound source relative to the user in a unit time, and the user may be a user using AR glasses. The relative movement direction of the sound source may refer to: the relative movement direction of the sound source and the user. If the relative movement direction of the sound source is a positive value, it may represent that the sound source is moving in a direction relatively far away from the user. If the relative movement direction of the sound source is a negative value, it may represent that the sound source is moving in a direction relatively close to the user. If the value of the relative movement direction of the sound source is 0, it may represent that the relative movement directions of the sound source and the user are consistent. The relative displacement change of the sound source may refer to: the relative displacement change between the sound source and the user. If the relative displacement change of the sound source is a positive value, it may represent that the relative displacement between the sound source and the user becomes larger, and the distance between the sound source and the user is increasing. If the relative displacement change of the sound source is a negative value, it may represent that the relative displacement between the sound source and the user becomes smaller, and the distance between the sound source and the user is shrinking. If the value of the relative displacement change of the sound source is 0, it may represent that the relative displacement between the sound source and the user has not changed, and the distance between the sound source and the user has not changed.

[0081] The collected audio data may include audio data of multiple sound sources, and for each sound source, this embodiment can determine the sound source category, sound source relative speed, sound source relative movement direction and sound source relative displacement change of the sound source by analyzing the audio data of the sound source. In addition, before analyzing the audio data, this embodiment can perform normalization, noise reduction, signal amplification and other pre-processing on the audio data to improve the data quality of the audio data. Specifically, this embodiment combines the pre-processed audio data with the Doppler effect of audio changes, captures the changes in the frequency spectrum of the audio data over time, analyzes the audio data through the CNN model, predicts the dynamic trend of audio changes, and comprehensively predicts the results of the above two methods to obtain the final sound source relative speed, sound source relative movement direction and sound source relative displacement change of the sound source. The specific process can be shown as steps one to four:

[0082] Step 1: Convert the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency features into two-dimensional matrix forms respectively, wherein the rows of the two-dimensional matrix are set as the data category dimension and the columns of the matrix are set as the time dimension;

[0083] Step 2: Input the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature in the form of a two-dimensional matrix into the convolutional neural network CNN model, and obtain the relative speed of the first sound source, the relative movement direction of the first sound source and the relative displacement change of the first sound source output by the CNN model;

[0084] Step 3: Input the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency characteristics in the form of a two-dimensional matrix into the Doppler effect model to obtain the relative speed of the second sound source, the relative movement direction of the second sound source and the relative displacement change of the second sound source output by the Doppler effect model;

[0085] Step 4: Perform weighted calculation on the relative speed of the first sound source, the relative movement direction of the first sound source and the relative displacement change of the first sound source, and the relative speed of the second sound source, the relative movement direction of the second sound source and the relative displacement change of the second sound source to obtain the relative speed of the sound source, the relative movement direction of the sound source and the relative displacement change of the sound source.

[0086] Among them, after the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature are respectively converted into a two-dimensional matrix form, specifically, in the two-dimensional matrix, the column can be set as the time dimension, each column can represent a time point or time period, and the row can be set as the corresponding data category dimension, such as in the two-dimensional matrix of the audio spectrum signal, the row can be set as the frequency dimension, and in the two-dimensional matrix of the sound source arrival angle signal, the row can be set as the arrival angle. Therefore, an element in the two-dimensional matrix can represent the value of the audio signal at a time.

[0087] In this embodiment, the CNN model can set one or more feature extraction layers according to actual needs to extract the features of the two-dimensional matrix. In addition, this embodiment can use a manually annotated data set as a pre-training data set to train the set CNN model to predict the ability of the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source. Specifically, after the audio data is converted into a matrix sequence representation in this embodiment, the matrix sequence can be input into the convolutional neural network CNN model, and the feature extraction is first performed through the feature extraction layer of the CNN model, and then the extracted features are mapped to the target variables (relative speed of the sound source, relative movement direction of the sound source, and relative displacement change of the sound source) through the fully connected layer of the CNN model, and finally the first sound source relative speed, the first sound source relative movement direction, and the first sound source relative displacement change are output through the output layer of the CNN model through the regression task.

[0088] The Doppler effect may refer to the phenomenon that the frequency of the wave received by the receiver changes when there is relative motion between the wave source (such as a sound wave source, a light wave source, etc.) and the receiver (such as an observer, a receiver, etc.). The Doppler effect model in this embodiment may be a model that describes the change in the frequency or wavelength of the sound source when the sound source and the user move relative to each other. In this embodiment, the audio data acquired by the microphone array may be directly input into the Doppler effect model, and the Doppler effect model directly calculates and outputs the second sound source relative speed, the second sound source relative motion direction, and the second sound source relative displacement change of the sound source. Among them, the second sound source relative speed may refer to the speed of the sound source relative to the microphone array, the second sound source relative motion direction may be obtained by calculating the change of the sound source arrival angle signal over time, and the second sound source relative displacement change may be obtained by calculating the integral of the second sound source relative speed and time.

[0089] After the present embodiment obtains the first sound source relative speed, the first sound source relative motion direction and the first sound source relative displacement change output by the CNN model and the second sound source relative speed, the second sound source relative motion direction and the second sound source relative displacement change output by the Doppler model, the output result of the CNN model and the output result of the Doppler model can be combined to obtain the sound source relative speed, the sound source relative motion direction and the sound source relative displacement change. Specifically, the present embodiment uses a weighted calculation method to combine the output results of the CNN model and the output results of the Doppler model, and after testing the test data set, when the weight of the output result of the CNN model is 0.35 and the weight of the output result of the Doppler model is 0.65, there are better results in the test data set, and the result of the comprehensive calculation is close to the result in the test data set.

[0090] S13. Determine the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source.

[0091] Among them, the color of the sound source image can be the color rendered when the sound source image is displayed on the display screen of the AR glasses, and the color of the sound source image can indicate the degree of danger of the sound source to the user. The visual change mode of the sound source image can be the dynamic effect displayed by the sound source when the image is displayed on the display screen of the AR glasses, and the visual change mode of the sound source image can indicate the reminder mode of the sound source on the display device of the AR glasses. The color and visual change mode of the sound source image need to be determined by calculating the sound source category, the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source. Specifically, this embodiment first determines the danger level of the sound source according to the sound source category. The danger level can include low risk, medium risk and high risk. The sound source category is the sound source classification label ID. The danger level corresponding to the sound source classification label ID can be set in advance and can be directly obtained through query. The correspondence between the audio classification label, the sound source classification label ID and the danger level can be shown in Table 2:

[0092] Table 2

[0093]

[0094] The relative movement of the sound source with respect to the user is then determined based on the relative speed of the sound source and the relative movement direction of the sound source. Specifically, whether the sound source moves relative to the user and the speed of the movement are determined based on the values ​​of the relative speed of the sound source and the relative movement direction of the sound source.

[0095] Finally, the color and visual change mode of the sound source image are determined based on the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source. Among them, a color can be determined according to the danger level, a color can be determined according to the relative movement of the sound source, and a color can be determined according to the relative displacement change of the sound source. The three determined colors are combined or selected to determine the final color of the sound source image, and the visual change mode is the same.

[0096] Specifically, in this embodiment, the color of the sound source image is determined according to the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source. The specific logic may be as follows:

[0097] When the sound source does not move relative to the source or the relative displacement of the sound source remains unchanged, the color is determined as follows: low risk is a safety color, medium risk and high risk are both the first degree of warning colors;

[0098] When the sound source moves relative to the source and the relative displacement of the sound source gradually increases, the color is determined as follows: low risk is a safety color, medium risk is a first-degree warning color, and high risk is a second-degree warning color;

[0099] When the sound source moves relative to the user and the relative displacement of the sound source gradually decreases, the color is determined as follows: according to the relative speed of the sound source and the current distance from the user, low risk is one of the first degree warning color, the second degree warning color and the third degree warning color, medium risk is one of the second degree warning color and the third degree warning color, and high risk is one of the second degree warning color and the third degree warning color;

[0100] Among them, the danger level represented by the first degree is lower than the danger level of the second degree warning color, and the danger level represented by the second degree warning color is lower than the danger level of the third degree warning color.

[0101] Among them, the safety color is the color set in the program to represent safety, and the warning color is the color set in the program to represent warning. In this embodiment, the safety color is green, the first degree of warning color is yellow, the second degree of warning color is orange, and the third degree of warning color is red.

[0102] When the sound source is first identified, if the sound source does not move relative to the source or the relative displacement of the sound source remains unchanged, the relative motion direction of the sound source can be expressed as 0 ( ) or the relative displacement change of the sound source is 0 ( ), then the relative movement direction of the sound source and the user is consistent or the distance between the sound source and the user has not changed, and the danger level to the user is low. Therefore, the sound source image of the low-risk level can be rendered in green, while the medium-risk and high-risk levels have certain dangers themselves. Even if there is no movement or the distance between the sound source and the user has not changed, they should be rendered in warning colors. Therefore, the sound source images of the medium-risk and high-risk levels can be rendered in yellow.

[0103] When the sound source is first identified, the sound source moves relative to the source and the relative displacement of the sound source gradually increases, which means that the relative movement direction of the sound source is 0 ( ) or the value of the relative displacement change of the sound source is 0 ( ), then the sound source is moving in a direction relatively far away from the user, and the distance between the sound source and the user is increasing, but because the sound source has moved, the degree of danger has increased. Therefore, the image of the sound source with a low risk level can be rendered in green, and the image of the sound source with a medium risk level and the image of the sound source with a high risk level can be rendered in yellow and orange respectively according to the risk level.

[0104] When the sound source is first identified, the sound source moves relative to the source and the relative displacement of the sound source gradually decreases, which means that the relative movement direction of the sound source is 0 ( ) or the value of the relative displacement change of the sound source is 0 ( ), then at this time the sound source is moving in a direction relatively close to the user, and the distance between the sound source and the user is shrinking, so the sound source is not only moving but also gradually approaching the user. Therefore, the danger level of the sound source to the user has increased again, and the low-risk sound source image can be upgraded to a warning color, while the medium-risk sound source image and the high-risk sound source image can both be upgraded to a warning color. Therefore, this embodiment needs to specifically consider the relative speed of the sound source and the current distance between the sound source and the user, determine the current danger level of the sound source to the user, and thus determine the rendering color of the sound source image. The specific logic can be as follows:

[0105] If the sound source moves relative to the source and the relative speed of the sound source is no greater than ( and ), and the distance between the user is not less than ( ), then the sound source image of the low-risk level can be rendered in yellow, the sound source image of the medium-risk level can be rendered in orange, and the sound source image of the high-risk level can be rendered in orange;

[0106] If the sound source moves relative to the source and the relative speed of the sound source is no greater than ( and ), and the distance between the user is less than ( ), the sound source image of the low-risk level can be rendered in orange, the sound source image of the medium-risk level can be rendered in red, and the sound source image of the high-risk level can be rendered in red;

[0107] If the sound source moves relative to the ( and ), and the distance between the user is not less than ( ), the sound source image of the low-risk level can be rendered in orange, the sound source image of the medium-risk level can be rendered in red, and the sound source image of the high-risk level can be rendered in red;

[0108] If the sound source moves relative to the ( and ), and the distance between the user is less than ( ), the sound source image of the low-risk level can be rendered in red, the sound source image of the medium-risk level can be rendered in red, and the sound source image of the high-risk level can be rendered in red.

[0109] The above situation is the color rendering logic of the sound source image when the sound source is first identified. When the sound source is identified again at the next time point, the color of the sound source image to be rendered can be determined again according to the above logic. If the color determined again according to the above logic is different from the current rendering color of the sound source image, the color of the sound source image can be changed, and optionally, the color of the sound source image can be changed in a gradual transition. If the color determined again according to the above logic is the same as the current rendering color of the sound source image, the current rendering color of the sound source image can be chosen not to be changed.

[0110] Specifically, in this embodiment, the visual change mode of the sound source image is determined according to the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source. The specific logic may be as follows:

[0111] When the sound source does not move relative to the sound source or the relative displacement of the sound source remains unchanged, the visual change is as follows: the sound source image is displayed at the display position;

[0112] When the sound source moves relatively and the relative displacement of the sound source gradually increases, the visual change mode is: the sound source image is displayed at the display position and the sound source image gradually shrinks;

[0113] When the sound source moves relative to the user and the relative displacement of the sound source gradually decreases, the visual changes are as follows: the sound source image is displayed at the low-risk display position and the sound source image gradually increases; the sound source image is displayed at the medium-risk and high-risk display positions and the sound source image gradually increases. Based on the relative speed of the sound source and the current distance from the user, it is determined whether to increase the flashing display effect.

[0114] Among them, when the sound source is first identified, if the sound source does not move relative to the source or the relative displacement of the sound source remains unchanged, the value of the relative movement direction of the sound source can be 0 ( ) or the value of the relative displacement change of the sound source is 0 ( ), then the relative movement direction of the sound source and the user is consistent or the distance between the sound source and the user has not changed. Therefore, the sound source image can be directly displayed at the display position, which can be the real mapping position of the sound source. For example, if the sound source is located in front of the left of the user, the display position of the sound source image can be a position in the upper left area of ​​the display screen of the AR glasses.

[0115] When the sound source is first identified, the sound source moves relative to the source and the relative displacement of the sound source gradually increases, which means that the relative movement direction of the sound source is 0 ( ) or the value of the relative displacement change of the sound source is 0 ( ), then the sound source is moving in a direction relatively far away from the user, and the distance between the sound source and the user is increasing, indicating that the sound source is getting farther and farther away from the user. In this case, a sound source image can be displayed at the display position and the sound source image gradually shrinks to reflect that the sound source is getting farther and farther away from the user.

[0116] When the sound source is first identified, the sound source moves relative to the source and the relative displacement of the sound source gradually decreases, which means that the relative movement direction of the sound source is 0 ( ) or the value of the relative displacement change of the sound source is 0 ( ), then the sound source is moving in a direction relatively close to the user, and the distance between the sound source and the user is shrinking, indicating that the sound source is getting closer to the user. Then the sound source image can be displayed at the display position and the sound source image gradually increases to reflect that the sound source is getting closer to the user. In some cases, the sound source is highly dangerous to the user, and it is difficult to clearly remind the user to pay attention to the sound source only by visual changes. Therefore, this embodiment can also determine whether to add a flashing display effect to the sound source image based on the relative speed of the sound source and the current distance between the sound source and the user. The specific judgment logic can be as follows:

[0117] If the sound source moves relative to the source and the relative speed of the sound source is no greater than ( and ), and the distance between the user is not less than ( ), then the low-risk level sound source image, the medium-risk level sound source image, and the high-risk level sound source image may not have the flickering display effect added;

[0118] If the sound source moves relative to the source and the relative speed of the sound source is no greater than ( and ), and the distance between the user is less than ( ), then the low-risk level sound source image, the medium-risk level sound source image, and the high-risk level sound source image may not have the flickering display effect added;

[0119] If the sound source moves relative to the ( and ), and the distance between the user is not less than ( ), then the low-risk level sound source image and the medium-risk level sound source image do not need to add the flickering display effect, and the high-risk level sound source image needs to add the flickering display effect.

[0120] If the sound source moves relative to the ( and ), and the distance between the user is less than ( ), then the low-risk sound source image does not need to add the flickering display effect, and the medium-risk sound source image and the high-risk sound source image both need to add the flickering display effect.

[0121] Similarly, the above situation is the logic for determining the visual change mode of the sound source image when the sound source is first identified. When the sound source is identified again at the next time point, the current visual change mode of the sound source image can also be determined again according to the above logic. If the visual change mode determined again according to the above logic is different from the current visual change mode of the sound source image, the visual change mode of the sound source image can be changed, and it can be selected whether to add a flashing display effect. Optionally, the visual change mode of the sound source image can be changed in the form of a display transition. If the visual change mode determined again according to the above logic is the same as the current visual change mode of the sound source image, it can be selected not to change the current visual change mode of the sound source image.

[0122] S14: Send the sound source image, the color of the sound source image, and the visual change mode to the display device of the augmented reality AR glasses, so that the sound source image with the color and visual change mode is displayed on the display device of the AR glasses.

[0123] Among them, after the sound source image corresponding to the sound source, the color of the sound source image and the visual change mode are determined according to the above content, they can be transmitted to the display screen of the AR glasses by the Bluetooth module for display. Figure 2 The display screen shown is a schematic diagram of a sound source. A user wearing AR glasses is walking on the street. There is a bicycle behind the user to the right, and a truck passes by in front of the user to the left. After the AR glasses collect audio data in the environment and analyze the audio data, they determine that the sound source category of one sound source is a truck. The virtual image of the sound source image can be a truck image and its color is yellow (it is farther away from the user and less dangerous to the user). The visual display method is: the sound source image is displayed at the display position (the upper left area of ​​the display screen, corresponding to the left front of the user). The sound source category of another sound source is a bicycle. The virtual image of the sound source image can be a bicycle image and its color is red (it is closer to the user and more dangerous to the user). The visual display method is: the sound source image is displayed at the display position (the lower right area of ​​the display screen, corresponding to the right rear of the user) and the sound source image is larger ( Figure 2 is a static picture, the actual dynamic effect is a gradual increase).

[0124] The embodiment of the present application provides a visual enhancement method, which obtains the sound source category, relative speed of the sound source, direction of movement of the sound source, and relative displacement change of the sound source corresponding to the audio data by analyzing the collected audio data, thereby determining the image display mode of the sound source on the display device of AR glasses according to the sound source category, relative speed of the sound source, relative movement direction of the sound source, and relative displacement change of the sound source. The sound source category can determine the sound source image corresponding to the sound source, and the sound source category, relative speed of the sound source, relative movement direction of the sound source, and relative displacement change of the sound source can determine the color and visual change mode of the sound source image. After determining the image display mode of the sound source on the display device of AR glasses, it can be sent to the display device of AR glasses, so that the sound source image with color and visual change mode is displayed on the display device of AR glasses. This method converts the analysis results of the audio data into the image display mode of the sound source on the display device of AR glasses, realizes the visual display of the sound source in the environment to remind the user, and the sound source also has corresponding color and visual changes on the display device of AR glasses, which can better remind the user. For the hearing-impaired who cannot hear the alarm or other important sounds, this embodiment can convert the sound into a visual signal and provide necessary safety warning information through AR glasses to remind the hearing-impaired.

[0125] Furthermore, this embodiment collects and analyzes multi-dimensional data such as timbre characteristics, frequency changes, amplitude changes, etc. of the sound source to provide early warning of environmental sounds. Compared with the defects of traditional traffic audio recognition models such as single indicators and one-sided prediction results, the practicality of the traffic audio environment warning in this embodiment is significantly improved.

[0126] In a possible implementation, the visual enhancement method may further include steps five to nine:

[0127] Step 5: Obtain a video containing the user's sign language movements, lip shapes, and facial expressions;

[0128] Step 6: Split the video into continuous image data.

[0129] Step 7: Analyze the sign language actions in the continuous image data to obtain a first predicted text corresponding to the sign language actions;

[0130] Step 8: Analyze the mouth shape and facial expression in the continuous image data to obtain a second predicted text corresponding to the mouth shape and facial expression;

[0131] Step nine: Send the target predicted text to the display device of the AR glasses, so that the target predicted text is displayed on the display device of the AR glasses, and the target predicted text is one of the first predicted text and the second predicted text.

[0132] Among them, this embodiment can collect a video containing the target person's sign language movements, mouth shape (lip reading) and facial expressions through the camera on the AR glasses. The target person can be: a person who is talking to the user or a person the user is looking at or observing. The target person's sign language movements can be: the target person's hand movements during speaking or demonstrating, which may not be limited to sign language. After the camera collects the video data, it is transmitted to the data processing part by the Bluetooth module for data processing.

[0133] The data processing part first performs preprocessing operations such as noise reduction and signal amplification on the video data, and then divides the video data into continuous image data. One video frame can be an image, and then analyzes the continuous image data through the CNN model to obtain the target predicted text. In addition to hand movements, the target person's lip expression and facial expressions play a certain role in communication or expression. Therefore, in the CNN model, the feature extraction processing of sign language movements and lip facial expressions in the continuous image data is performed separately to obtain the first predicted text and the second predicted text respectively. Finally, the output layer compares the first predicted text and the second predicted text to obtain the target predicted text.

[0134] Furthermore, if the camera does not capture the target person's sign language movements, mouth shape (lip reading) and facial expressions, but captures the target person's audio data through the microphone, the audio data can be transmitted to the data processing part through the Bluetooth module. The data processing part first performs preprocessing operations such as noise reduction and signal amplification, and then converts the preprocessed audio data into a matrix form, uses deep learning technology to perform matrix fusion and feature extraction, and outputs the prediction results of the fused feature matrix through the classifier. Finally, the prediction results are processed by natural language compilation to generate a coherent text expression sentence, which is then transmitted to the display screen by the Bluetooth module for information projection.

[0135] Specifically, for sign language actions, when capturing the features of sign language actions in continuous image data, in the feature extraction layer of the CNN model, the bidirectional attention mechanism of the CNN+BiLSTM fusion model (a deep learning model that integrates convolutional neural networks and bidirectional long short-term memory networks) can be used to capture the subtle features of sign language actions in continuous image data, comprehensively generate feature vectors, and combine the feature vectors to generate a feature matrix. After display segmentation and label extraction of the feature matrix by CTC (Connectionist Temporal Classification), the first predicted text is obtained. Among them, BiLSTM (Bidirectional Long Short-Term Memory) is a special recurrent neural network. BiLSTM can be composed of two independent LSTM layers: one is a forward LSTM that processes data step by step from the beginning to the end of the sequence; the other is a reverse LSTM that processes data from the end to the beginning of the sequence. Therefore, the two LSTM layers capture information from front to back and from back to front in the sequence respectively to improve the performance of the model when processing sequence data. Among them, due to the differences in sign language in different regions, the CNN+BiLSTM fusion model is trained using the common sign language datasets of each region as the pre-training dataset.

[0136] For lip shape and facial expression, when capturing the features of lip shape and facial expression in continuous image data, this embodiment can use the CNN+Xception model in the feature extraction layer of the CNN model to analyze the changes in lip shape (lip reading), eyebrows, eyes and other parts in the continuous image data, extract relevant features and generate a feature matrix, and then map the feature matrix to the text through the fully connected layer to obtain the second predicted text. When performing the task of analyzing lip shape and facial expression, the CNN+Xception model is pre-trained by the CMLR data set. Due to the characteristics of the deep separable convolution in the Xception model, the model calculation amount of the CNN model can be reduced and the feature extraction ability of the model can be improved. Among them, the CMLR (Chinese Mandarin LipReading) data set is a lip reading recognition data set specifically for Mandarin Chinese. Of course, foreign lip shape data sets can also be used for training to improve the model's recognition of foreign spoken language. The Xception model is a deep convolutional neural network architecture that uses deep separable convolution to achieve feature extraction. Deep separable convolution can decompose the standard convolution operation into two steps: depth convolution and pointwise convolution. The depth convolution applies a convolution kernel on each input channel to reduce the amount of computation and number of parameters of the model, and the point-by-point convolution uses a 1×1 convolution kernel for linear combinations between channels to increase the representation capability of the model.

[0137] When the output layer synthesizes the first predicted text and the second predicted text to obtain the target predicted text, one of them can be selected as the target predicted text by comparison, or the first predicted text and the second predicted text can be synthesized by weighted calculation or the like to obtain the target predicted text. After obtaining the target predicted text, it is transmitted to the display screen of the AR glasses via the Bluetooth module, and the display screen projects the information of the target predicted text to assist communication between users or help users understand the expression content of the target person.

[0138] This embodiment uses a deep learning algorithm to mine features from multiple data sources (sign language movements, mouth shapes (lip readings) and facial expressions) when identifying the target person's expression to achieve text prediction, optimize the singleness and limitations of traditional sign language model indicators, and effectively improve the prediction effect. In addition, it can help users who do not master sign language (here, it can refer to dumb language) understand the meaning of sign language expression and promote communication between users.

[0139] A visual enhancement method provided by an embodiment of the present application is introduced above, and a system for executing the above visual enhancement method will be introduced below.

[0140] See also Figure 3 , Figure 3This is a schematic diagram of the structure of a visual enhancement system provided in an embodiment of the present application. Figure 3 As shown, the visual enhancement system may include:

[0141] An acquisition unit 100 is used to acquire collected audio data;

[0142] The analysis unit 110 is used to analyze the audio data to obtain the sound source category, the relative speed of the sound source, the relative movement direction of the sound source and the relative displacement change of the sound source corresponding to the audio data;

[0143] An image recognition unit 120, configured to determine a corresponding sound source image according to a sound source category;

[0144] A display determination unit 130, for determining the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source;

[0145] The sending unit 140 is used to send the sound source image, the color and visual change mode of the sound source image to the display device of the augmented reality AR glasses, so that the sound source image with color and visual change mode is displayed on the display device of the AR glasses.

[0146] In a possible implementation, the audio data includes: audio spectrum signal, sound source arrival angle signal, audio amplitude difference, time difference and time-frequency feature. The time-frequency feature is: after the audio spectrum signal is framed, the feature is extracted by short-time Fourier transform analysis.

[0147] The relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source corresponding to the audio data obtained in the analysis unit 110 can be specifically configured as follows:

[0148] The audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency characteristics are respectively converted into two-dimensional matrix forms, wherein the rows of the two-dimensional matrix are set as the data category dimension, and the columns of the matrix are set as the time dimension; the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency characteristics in the form of a two-dimensional matrix are input into a convolutional neural network CNN model to obtain the first sound source relative speed, the first sound source relative movement direction and the first sound source relative displacement change output by the CNN model; the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency characteristics in the form of a two-dimensional matrix are input into a Doppler effect model to obtain the second sound source relative speed, the second sound source relative movement direction and the second sound source relative displacement change output by the Doppler effect model; the first sound source relative speed, the first sound source relative movement direction and the first sound source relative displacement change are weightedly calculated with the second sound source relative speed, the second sound source relative movement direction and the second sound source relative displacement change to obtain the sound source relative speed, the sound source relative movement direction and the sound source relative displacement change.

[0149] In a possible implementation, the audio data includes: an audio spectrum signal,

[0150] The analysis unit 110 analyzes the audio data to obtain the sound source category of the sound source corresponding to the audio data, which can be specifically configured as follows:

[0151] The audio spectrum signal is framed, and the Mel-frequency cepstrum coefficients are used to convert the framed audio spectrum signal into a two-dimensional matrix, the rows of the two-dimensional matrix are set as the frequency dimension, and the columns of the two-dimensional matrix are set as the time dimension; the audio spectrum signal in the form of a two-dimensional matrix is ​​input into a convolutional neural network (CNN) model to obtain the sound source classification label ID output by the convolutional neural network (CNN) model, and the sound source classification label ID is used as the sound source category.

[0152] In a possible implementation, the sound source category is a sound source classification label ID, and the image recognition unit 120 may be specifically configured as follows:

[0153] The virtual shape of the object corresponding to the sound source classification label ID is used as the sound source image corresponding to the sound source category, and the virtual shape of the object is a pedestrian, an animal, or a vehicle.

[0154] In one possible implementation, the color can indicate the degree of danger of the sound source to the user, and the visual change mode can indicate the reminder mode of the sound source on the display device of the AR glasses.

[0155] The display determination unit 130 may include:

[0156] A level determination subunit, used to determine the danger level of the sound source according to the sound source category;

[0157] A motion analysis subunit, used to determine the relative motion of the sound source with respect to the user based on the relative speed and relative motion direction of the sound source;

[0158] The visual determination subunit is used to determine the color and visual change mode of the sound source image according to the danger level of the sound source, the relative movement of the sound source and the relative displacement change of the sound source.

[0159] In a possible implementation, the danger level of the sound source includes low danger, medium danger and high danger.

[0160] The visual determination subunit determines the color of the sound source image according to the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source. Specifically, the configuration may be as follows:

[0161] When the sound source does not move relative to the user or the relative displacement of the sound source remains unchanged, the color is determined as follows: low risk is a safety color, and medium risk and high risk are both first-degree warning colors; when the sound source moves relative to the user and the relative displacement of the sound source gradually increases, the color is determined as follows: low risk is a safety color, medium risk is a first-degree warning color, and high risk is a second-degree warning color; when the sound source moves relative to the user and the relative displacement of the sound source gradually decreases, the color is determined as follows: according to the relative speed of the sound source and the current distance from the user, low risk is one of the first-degree warning color, the second-degree warning color, and the third-degree warning color, medium risk is one of the second-degree warning color and the third-degree warning color, and high risk is one of the second-degree warning color and the third-degree warning color;

[0162] Among them, the danger level represented by the first degree warning color is lower than the danger level of the second degree warning color, and the danger level represented by the second degree warning color is lower than the danger level of the third degree warning color.

[0163] In a possible implementation, the danger level of the sound source includes low danger, medium danger and high danger.

[0164] The visual determination subunit determines the visual change mode of the sound source image according to the danger level of the sound source, the relative movement of the sound source, and the relative displacement change of the sound source. Specifically, the configuration may be as follows:

[0165] When the sound source does not move relative to the sound source or the relative displacement of the sound source remains unchanged, the visual change is: the sound source image is displayed at the display position; when the sound source moves relative to the sound source and the relative displacement of the sound source gradually increases, the visual change is: the sound source image is displayed at the display position and the sound source image gradually shrinks; when the sound source moves relative to the sound source and the relative displacement of the sound source gradually decreases, the visual change is: the sound source image is displayed at the display position for low risk and the sound source image gradually increases, the sound source image is displayed at the display position for medium risk and high risk and the sound source image gradually increases, and according to the relative speed of the sound source and the current distance between the sound source and the user, it is determined whether to increase the flashing display effect.

[0166] In a possible implementation, the visual enhancement system may further include a recognition unit, which may be specifically configured as follows:

[0167] A video containing the user's sign language movements, lip shapes and facial expressions is acquired; the video is split into continuous image data, the sign language movements in the continuous image data are analyzed, and a first predicted text corresponding to the sign language movements is obtained; the lip shapes and facial expressions in the continuous image data are analyzed, and a second predicted text corresponding to the lip shapes and facial expressions is obtained; the target predicted text is sent to a display device of the AR glasses, so that the target predicted text is displayed on the display device of the AR glasses, and the target predicted text is one of the first predicted text and the second predicted text.

[0168] The present application also provides an electronic device in an embodiment. Figure 4 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0169] like Figure 4 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0170] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a memory card, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0171] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the visual enhancement methods provided in the embodiments of the present application.

[0172] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any visual enhancement method provided in the embodiment of the present application.

[0173] It should also be noted that the system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0174] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0175] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0176] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0177] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0178] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0179] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A visual enhancement method, characterized in that: The visual enhancement method comprises: Get the collected audio data; Analyze the audio data to obtain a sound source category, a relative speed of the sound source, a relative movement direction of the sound source, and a relative displacement change of the sound source corresponding to the audio data; Determine a corresponding sound source image according to the sound source category; Determine the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source and the relative displacement change of the sound source; The sound source image, the color of the sound source image and the visual change mode are sent to a display device of augmented reality AR glasses, so that the sound source image with the color and the visual change mode is displayed on the display device of the AR glasses.

2. The visual enhancement method according to claim 1, characterized in that: The audio data includes: audio spectrum signal, sound source arrival angle signal, audio amplitude difference, time difference and time-frequency feature, wherein the time-frequency feature is: the feature extracted by short-time Fourier transform analysis after the audio spectrum signal is framed, Analyzing the audio data to obtain the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source corresponding to the audio data, including: Convert the audio spectrum signal, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature into two-dimensional matrix forms respectively, wherein the rows of the two-dimensional matrix are set as data category dimensions and the columns of the matrix are set as time dimensions; Input the audio spectrum signal in the form of a two-dimensional matrix, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature into a convolutional neural network (CNN) model, and obtain the first sound source relative speed, the first sound source relative motion direction and the first sound source relative displacement change output by the CNN model; Inputting the audio spectrum signal in a two-dimensional matrix form, the sound source arrival angle signal, the audio amplitude difference, the time difference and the time-frequency feature into a Doppler effect model, and obtaining the second sound source relative speed, the second sound source relative movement direction and the second sound source relative displacement change output by the Doppler effect model; The first sound source relative speed, the first sound source relative movement direction and the first sound source relative displacement change are weightedly calculated with the second sound source relative speed, the second sound source relative movement direction and the second sound source relative displacement change to obtain the sound source relative speed, the sound source relative movement direction and the sound source relative displacement change.

3. The visual enhancement method according to claim 1, characterized in that: The audio data includes: an audio spectrum signal, Analyzing the audio data to obtain a sound source category of a sound source corresponding to the audio data includes: The audio spectrum signal is framed, and the framed audio spectrum signal is converted into a two-dimensional matrix using Mel-frequency cepstrum coefficients, wherein the rows of the two-dimensional matrix are set as frequency dimensions, and the columns of the two-dimensional matrix are set as time dimensions; The audio spectrum signal in the form of a two-dimensional matrix is ​​input into a convolutional neural network (CNN) model to obtain a sound source classification label ID output by the CNN model, and the sound source classification label ID is used as the sound source category.

4. The visual enhancement method according to claim 1, characterized in that: The sound source category is a sound source classification label ID, and determining the corresponding sound source image according to the sound source category includes: The virtual shape of the object corresponding to the sound source classification tag ID is used as the sound source image corresponding to the sound source category, and the virtual shape of the object is a pedestrian, an animal, or a vehicle.

5. The visual enhancement method according to claim 1, characterized in that: The color indicates the degree of danger of the sound source to the user, the visual change mode indicates the reminder mode of the sound source on the display device of the AR glasses, and the color and visual change mode of the sound source image are determined according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source, including: determining the danger level of the sound source according to the sound source category; Determining the relative movement of the sound source with respect to the user according to the relative speed of the sound source and the relative movement direction of the sound source; The color and the visual change mode of the sound source image are determined according to the danger level of the sound source, the relative movement of the sound source and the relative displacement change of the sound source.

6. The visual enhancement method according to claim 5, characterized in that: The danger level of the sound source includes low danger, medium danger and high danger. The color of the sound source image is determined according to the danger level of the sound source, the relative movement of the sound source and the relative displacement change of the sound source, including: When the sound source does not move relative to the sound source or the relative displacement of the sound source remains unchanged, the color is determined as follows: the low risk is a safety color, and the medium risk and the high risk are both the first degree of warning colors; When the sound source moves relative to the sound source and the relative displacement of the sound source gradually increases, the color is determined in the following manner: the low risk is a safety color, the medium risk is a warning color of the first degree, and the high risk is a warning color of the second degree; When the sound source moves relative to the user and the relative displacement of the sound source gradually decreases, the color is determined as follows: according to the relative speed of the sound source and the current distance between the sound source and the user, the low risk is one of the first degree of warning color, the second degree of warning color, and the third degree of warning color; the medium risk is one of the second degree of warning color and the third degree of warning color; and the high risk is one of the second degree of warning color and the third degree of warning color; The danger level represented by the first level of warning color is lower than the danger level represented by the second level of warning color, and the danger level represented by the second level of warning color is lower than the danger level represented by the third level of warning color.

7. The visual enhancement method according to claim 5, characterized in that: The danger level of the sound source includes low danger, medium danger and high danger. The visual change mode of the sound source image is determined according to the danger level of the sound source, the relative movement of the sound source and the relative displacement change of the sound source, including: When the sound source does not move relative to the sound source or the relative displacement of the sound source remains unchanged, the visual change mode is: displaying the sound source image at the display position; When the sound source moves relatively and the relative displacement of the sound source gradually increases, the visual change mode is: the sound source image is displayed at the display position and the sound source image is gradually reduced; When the sound source undergoes relative motion and the relative displacement of the sound source gradually decreases, the visual change mode is: the low-risk sound source image is displayed at the display position and the sound source image gradually increases, the medium-risk and high-risk sound source images are displayed at the display position and the sound source image gradually increases, and according to the relative speed of the sound source and the current distance between the sound source and the user, it is determined whether to increase the flashing display effect.

8. The visual enhancement method according to claim 1, characterized in that: The visual enhancement method further comprises: Obtain a video containing the user's sign language movements, mouth shapes, and facial expressions; Splitting the video into continuous image data, Analyze the sign language action in the continuous image data to obtain a first predicted text corresponding to the sign language action; Analyze the lip shape and facial expression in the continuous image data to obtain a second predicted text corresponding to the lip shape and the facial expression; The target predicted text is sent to a display device of the AR glasses, so that the target predicted text is displayed on the display device of the AR glasses, and the target predicted text is one of the first predicted text and the second predicted text.

9. A visual enhancement system, characterized in that: The visual enhancement system comprises: An acquisition unit, used for acquiring collected audio data; An analysis unit, configured to analyze the audio data to obtain a sound source category, a relative speed of the sound source, a relative movement direction of the sound source, and a relative displacement change of the sound source corresponding to the audio data; An image recognition unit, used to determine a corresponding sound source image according to the sound source category; A display determination unit, configured to determine the color and visual change mode of the sound source image according to the sound source category, the relative speed of the sound source, the relative movement direction of the sound source, and the relative displacement change of the sound source; A sending unit is used to send the sound source image, the color of the sound source image and the visual change mode to the display device of the augmented reality AR glasses, so that the sound source image with the color and the visual change mode is displayed on the display device of the AR glasses.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the visual enhancement method as described in any one of claims 1 to 8.

Images