Semantic recognition method and related device

Abstract

The embodiment of the invention provides a semantic recognition method and a related device, and relates to the technical field of terminals. The method comprises the following steps: acquiring data such as vibration frequency of a vocal cord of a user based on a radar of the electronic equipment, and analyzing the data acquired by the radar, so as to obtain semantics which the user wants to express. According to the technical scheme, the radar is adopted to detect vocal cord vibration, and good privacy and transmission capacity are achieved, so that the electronic equipment does not completely depend on voice signal or mouth shape processing, the semantics of a user can be accurately recognized even in a noisy environment or a dark environment or an environment with a shielding object, the call quality is improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a semantic recognition method and related devices. Background Art

[0002] In some scenarios, electronic devices can obtain the user's voice and implement corresponding functions, such as call scenarios, live broadcast scenarios, etc.

[0003] However, due to interference from environmental factors, the user's voice received by the electronic device may be mixed with other noises; or, because the user's voice may be relatively small, the electronic device cannot clearly obtain the user's voice, resulting in reduced call quality. Summary of the invention

[0004] The semantic recognition method and related devices provided in the embodiments of the present application can obtain data such as the vibration frequency of the user's vocal cords based on the radar of the electronic device, and analyze the data obtained by the radar to obtain the semantics that the user wants to express, improve call quality, and thus enhance the user experience.

[0005] In a first aspect, a semantic recognition method provided by an embodiment of the present application is applied to an electronic device, the electronic device including a radar, and the method includes:

[0006] In the process of collecting the user's audio and video data, the vibration frequency of the user's vocal cords is obtained based on the radar; a first sound recognition result for semantic recognition of the vibration frequency of the vocal cords is obtained; a target sound recognition result of the audio and video data is obtained based on the first sound recognition result; the target sound recognition result is the first sound recognition result, or the target sound recognition result is obtained by processing the first sound recognition result and the second audio recognition result, or the target sound recognition result is obtained by processing the first sound recognition result, the second audio recognition result and the third image recognition result, wherein the second audio recognition result is obtained by semantic recognition of the audio in the audio and video data, and the third image recognition result is obtained by lip recognition of the user in the audio and video data; the target sound recognition result is displayed. In this way, the use of radar to detect vocal cord vibration has good privacy and transmission capabilities, and the electronic device does not have to rely entirely on the processing of voice signals or lip shapes, so that even in noisy, dimly lit or obstructed environments, the user's semantics can be more accurately recognized.

[0007] In a possible implementation, before obtaining the first sound recognition result of semantically identifying the vibration frequency of the vocal cords, it also includes: preprocessing the vibration frequency of the vocal cords in the distance dimension to obtain the time-frequency diagram corresponding to the vibration frequency of the vocal cords; and semantically identifying the time-frequency diagram corresponding to the vibration frequency of the vocal cords. In this way, the radar can search for the vibration frequency of the vocal cords in different distance dimensions and accumulate the searched information, so as to reduce the dimension of the data and obtain a relatively complete vibration frequency of the vocal cords, which is convenient for further extraction of useful signals in the received signal.

[0008] In a possible implementation, a radar is used to obtain an electromagnetic wave signal, which includes the vibration frequency of the vocal cords, and the vibration frequency of the vocal cords is preprocessed in the distance dimension, including: determining the first position with the maximum energy in the electromagnetic wave signal within the radar search range; searching the second position in the direction of the boundary of the radar search range with the first position as the center; if the Rayleigh entropy of the second position is less than a preset value, the signal corresponding to the second position is retained; or, if the Rayleigh entropy of the second position is greater than or equal to the preset value, the signal corresponding to the second position is not retained; and a time-frequency diagram is output based on the retained signal, and the time-frequency diagram includes a time-frequency diagram corresponding to the vibration frequency of the vocal cords. In this way, the radar can select the place with the strongest energy in the received signal as the target center point, and search around the target center point, so as to obtain relatively complete useful information, which is convenient for further extraction of the vibration frequency signal of the vocal cords in the received signal.

[0009] In a possible implementation, before determining the first position of the maximum energy in the electromagnetic wave signal, it also includes: performing time-frequency transformation on different distance dimensions of the electromagnetic wave signal to obtain a time-frequency distribution diagram STFT of each distance dimension; determining a preset value based on the Rayleigh entropy of FTFR, the preset value includes the average value of the sum of the Rayleigh entropies of some or all FTFRs. In this way, after the time-frequency transformation processing, the target signal has better local focusing in time and frequency, and the distinction between the target signal and clutter is more obvious, so that a clearer and more concentrated energy distribution of the target signal can be obtained.

[0010] In a possible implementation, before the distance dimension preprocessing of the vibration frequency of the vocal cords, it also includes: performing one or more of the following processing on the electromagnetic wave signal: moving target detection, constant false alarm detection, multipath suppression processing, and time-frequency focusing processing; wherein the moving target detection is used to filter the signal with zero frequency or less than the preset frequency value in the electromagnetic wave signal, the constant false alarm detection is used to filter the signal with low frequency energy in the electromagnetic wave signal, the multipath suppression processing is used to filter the echo signal in the electromagnetic wave signal, the echo signal includes the signal reflected back when the radar's transmission signal encounters an obstacle, and the time-frequency focusing processing is used to enhance the time-frequency focusing ability of the vibration frequency of the vocal cords in the electromagnetic wave signal. In this way, the moving target detection can filter the signal with zero frequency or a value near the frequency period in the electromagnetic wave signal, thereby highlighting the target signal. The constant false alarm detection can filter the signal with low frequency energy in the electromagnetic wave signal, suppress the clutter in the received signal, and thus more accurately identify the target signal. The multipath suppression processing can filter the echo signal in the electromagnetic wave signal and reduce the interference of the echo signal on the target signal. Time-frequency focusing processing can utilize the time-frequency focusing difference between the target signal and the clutter to enhance the time-frequency focusing ability of the vibration frequency of the vocal cords in the electromagnetic wave signal, thereby improving the time-frequency focusing ability of the target signal and further enhancing the time-frequency extraction effect of the target signal.

[0011] In a possible implementation, before obtaining the first sound recognition result of semantic recognition of the vibration frequency of the vocal cords, it also includes: displaying a first interface, the first interface including one or more of the following options: a first option, a second option, a third option, and a fourth option; wherein the first option is used to indicate the real-time requirement for semantic recognition of the vibration frequency of the vocal cords, the first option includes a first level and a second level, and the real-time requirement of the first level is higher than that of the second level; the second option is used to indicate the accuracy requirement for semantic recognition of the vibration frequency of the vocal cords, the second option includes a third level and a fourth level, and the accuracy requirement of the third level is higher than that of the fourth level; the third option is used to indicate the accuracy requirement for semantic recognition of the vibration frequency of the vocal cords The privacy requirement for semantic recognition of vibration frequency, the third option includes the fifth level and the sixth level, and the privacy requirement of the fifth level is higher than that of the sixth level; the fourth option is used to indicate the traffic consumption requirement for semantic recognition of the vibration frequency of the vocal cords, the fourth option includes the seventh level and the eighth level, and the traffic consumption requirement of the seventh level is higher than that of the eighth level; when the first level, the fourth level, the fifth level or the seventh level is selected, the second level, the third level, the sixth level and / or the eighth level are in an unselectable state; when the second level, the third level, the sixth level and / or the eighth level are selected, the first level, the fourth level, the fifth level and / or the seventh level are in an unselectable state. In this way, the selection of each option in the first interface may conflict, for example, both fast real-time performance and strict accuracy are required. In order to reduce conflicts between options, when a user selects an option, the electronic device can set other options that conflict with the option to an unselectable state, such as graying them out, so that the user cannot select conflicting options at the same time, thereby improving the rationality of the execution logic of the electronic device.

[0012] In one possible implementation, when a selection operation for the first level, the fourth level, the fifth level, or the seventh level is received, a first sound recognition result for semantically identifying the vibration frequency of the vocal cords is obtained, including: using a convolutional neural network to semantically identify the vibration frequency of the vocal cords to obtain the first sound recognition result. In this way, the electronic device does not need to upload the data to the cloud server, saving the traffic occupied by uploading data. Since local processing does not require additional time to upload to the cloud server, data processing is performed locally on the electronic device, and the recognition of the call content can be completed more quickly.

[0013] In a possible implementation, when a selection operation for the second level, the third level, the sixth level and / or the eighth level is received, a first sound recognition result for semantically identifying the vibration frequency of the vocal cords is obtained, including: uploading the vibration frequency of the vocal cords to a cloud server, and obtaining the first sound recognition result for semantically identifying the vibration frequency of the vocal cords from the cloud server, the first sound recognition result being obtained by the cloud server using a convolutional neural network to semantically identify the vibration frequency of the vocal cords. In this way, since the cloud server has a strong computing power and / or a strong storage capacity, the cloud server has a strong ability to process data, and can more accurately analyze and identify the user's speech content, thereby improving the user experience.

[0014] In a possible implementation, before obtaining the first sound recognition result for semantically identifying the vibration frequency of the vocal cords, it also includes: performing principal component analysis PCA on the vibration frequency of the vocal cords to extract characteristic data related to the recognition semantics, the characteristic data including one or more of the following: the highest frequency of vocal cord vibration, the lowest frequency of vocal cord vibration, the ratio of the highest frequency to the lowest frequency, and the duration of the vocal cord vibration frequency; obtaining the first sound recognition result for semantically identifying the vibration frequency of the vocal cords includes: obtaining the first sound recognition result for semantically identifying the characteristic data. In this way, PCA can filter out the more important features in the data, thereby reducing the amount of data, while retaining the information of the original data as much as possible. In this way, the characteristic data related to the user can be extracted, and the features unrelated to the user can be filtered, so as to extract useful information from the received signal more quickly.

[0015] In a second aspect, an embodiment of the present application provides a device for semantic recognition, which may be an electronic device, or a chip or chip system in an electronic device. The device may include a processing unit and a display unit. The processing unit is used to implement any method related to processing performed by the electronic device in the first aspect or any possible implementation of the first aspect. The display unit is used to implement any method related to display performed by the electronic device in the first aspect or any possible implementation of the first aspect. When the device is an electronic device, the processing unit may be a processor. The device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements the method described in the first aspect or any possible implementation of the first aspect. When the device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit in the electronic device located outside the chip (for example, a read-only memory, a random access memory, etc.).

[0016] Exemplarily, the processing unit is used to obtain the vibration frequency of the user's vocal cords; further used to obtain a first sound recognition result for semantic recognition of the vibration frequency of the vocal cords; and specifically further used to obtain a target sound recognition result of the audio and video data based on the first sound recognition result. The display unit is used to display the target sound recognition result.

[0017] In one possible implementation, the processing unit is used to preprocess the vibration frequency of the vocal cords in the distance dimension to obtain a time-frequency diagram corresponding to the vibration frequency of the vocal cords; and is also used to perform semantic recognition on the time-frequency diagram corresponding to the vibration frequency of the vocal cords.

[0018] In one possible implementation, the processing unit is used to determine a first position with maximum energy in an electromagnetic wave signal within a radar search range; it is also used to search a second position in a direction toward a boundary of the radar search range with the first position as the center; it is specifically used to retain a signal corresponding to the second position if the Rayleigh entropy of the second position is less than a preset value; or, if the Rayleigh entropy of the second position is greater than or equal to a preset value, not retain the signal corresponding to the second position; and it is also used to output a time-frequency diagram based on the retained signal.

[0019] In a possible implementation, the processing unit is used to perform time-frequency transformation on different distance dimensions of the electromagnetic wave signal to obtain a time-frequency distribution diagram STFT of each distance dimension; and is also used to determine a preset value based on the Rayleigh entropy of FTFR.

[0020] In a possible implementation, the processing unit is used to perform one or more of the following processing on the electromagnetic wave signal: moving target detection, constant false alarm detection, multipath suppression processing, and time-frequency focusing processing.

[0021] In a possible implementation, the display unit is used to display the first interface.

[0022] In one possible implementation, the processing unit is used to perform semantic recognition on the vibration frequency of the vocal cords using a convolutional neural network to obtain a first sound recognition result.

[0023] In a possible implementation, the processing unit is used to upload the vibration frequency of the vocal cords to a cloud server, and obtain a first sound recognition result of semantically identifying the vibration frequency of the vocal cords from the cloud server.

[0024] In a possible implementation, the processing unit is used to perform principal component analysis PCA on the vibration frequency of the vocal cords to extract characteristic data related to recognition semantics; and is also used to obtain a first sound recognition result of semantically recognizing the characteristic data.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory, wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in the first aspect or any possible implementation of the first aspect.

[0026] In a fourth aspect, the present application provides a chip or a chip system, which is applied to an electronic device, and the chip or chip system includes one or more processors and a communication interface, wherein the communication interface and at least one processor are interconnected through a line, and the one or more processors are used to call computer instructions so that the electronic device executes the method described in the first aspect or any possible implementation of the first aspect. Among them, the communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0027] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0028] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0030] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a radar for acquiring the vibration frequency of vocal cords provided in an embodiment of the present application;

[0034] Figure 4 A flowchart of a semantic recognition method provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of a radar function interface provided in an embodiment of the present application;

[0036] Figure 6 A flowchart of determining the correlation between a target signal and a suspected multipath signal provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of a time-frequency extraction range provided in an embodiment of the present application;

[0038] Figure 8 A flow chart of time-frequency extraction provided in an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of a semantic recognition method provided in an embodiment of the present application;

[0040] Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0042] 1. Terminology

[0043] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0044] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0045] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0046] 2. Electronic equipment

[0047] The electronic device of the embodiment of the present application may also be a terminal device in any form. For example, the electronic device may include: a mobile phone, a tablet computer, a PDA, a laptop computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, an electronic device in a 5G network or a public land mobile communication network (public land mobile communication network) to be evolved in the future. Mobile network, PLMN) and the like, the embodiments of the present application are not limited to this.

[0048] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0049] In addition, in the embodiments of the present application, the electronic device may also be an electronic device in an Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0050] The electronic devices in the embodiments of the present application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0051] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0052] For example, Figure 1 A schematic structural diagram of an electronic device is shown.

[0053] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0054] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may include hardware, software, or a combination of software and hardware.

[0055] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0056] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or is cyclically using. If the processor 110 needs to use the instruction or data again, it may be directly called from the above-mentioned memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system. For example, in an embodiment of the present application, the processor 110 may be used for processing radar data, determining real-time requirements, and identifying call semantics.

[0057] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0058] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data created during the use of the electronic device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. For example, in an embodiment of the present application, the internal memory 121 may be used to store radar data, and related codes for processing radar data, etc.

[0059] The camera 193 is used to capture static images or videos. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1. For example, in the embodiment of the present application, the camera can obtain the changes in the user's lip shape, thereby identifying the semantics corresponding to the lip shape.

[0060] Figure 2It is a software structure block diagram of the electronic device of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, the hardware abstraction layer (hardware adaptation layer, HAL), and the kernel layer.

[0061] The application layer can also be called the application layer, which can include a series of application packages. Figure 2 As shown, the application package may include applications such as phone, music, camera, etc. Applications may include system applications and third-party applications.

[0062] The application framework layer can also be called the Framework layer. The Framework layer can provide an application programming interface (API) and a programming framework for the application layer's applications. The Framework layer can include some predefined functions.

[0063] like Figure 2 As shown, the Framework layer may include an activity manager, a window manager, a resource manager, a notification manager, a content provider, and a view system, etc.

[0064] Android runtime includes core libraries and virtual machines. Android runtime is responsible for the control and management of the Android system.

[0065] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0066] The application layer and the framework layer run in a virtual machine. The virtual machine executes the java files of the application layer and the framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection. For example, in the embodiment of the present application, the virtual machine can be used to perform functions such as radar data collection and radar data analysis.

[0067] The system library can also be called the Native layer, which can include multiple functional modules, such as the media library, function library, and graphics processing library.

[0068] The hardware abstraction layer is an abstract structure between the kernel layer and the Android runtime. The hardware abstraction layer can be a package of hardware drivers, providing a unified interface for upper-layer applications to call.

[0069] The kernel layer is the layer between hardware and software. The kernel layer may include display drivers, camera drivers, audio drivers, etc.

[0070] It should be noted that the embodiments of the present application are only illustrated using the Android system as an example. In other operating systems (such as Windows system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the solutions of the present application can also be implemented.

[0071] In scenarios such as calls or live broadcasts, electronic devices may be disturbed by environmental factors, so that the received user voice may be mixed with other noises, reducing the call quality. Or, when making a call in a quiet environment, in order not to disturb others, the user's voice may be low, and the electronic device may not be able to clearly receive the user's voice, reducing the call quality.

[0072] In some implementations, in order to improve call quality, the electronic device can recognize the user's semantics based on the camera's lip recognition technology. However, in dimly lit scenes, or scenes where the user is wearing a mask or other obstructing objects, the camera cannot recognize the user's semantics based on the lip recognition technology, thereby affecting the accuracy of lip recognition.

[0073] In view of this, the semantic recognition method provided in the embodiment of the present application can obtain data such as the vibration frequency of the user's vocal cords based on the radar of the electronic device, and analyze the data obtained by the radar to obtain the semantics that the user wants to express. The use of radar to detect vocal cord vibration has good privacy and transmission capabilities. In this way, the electronic device does not have to rely entirely on the processing of voice signals or lip shapes, so that even in noisy, dimly lit or obstructed environments, it can more accurately recognize the user's semantics, improve call quality, and thus enhance user experience.

[0074] Take the call scenario as an example. Figure 3 A schematic diagram of radar acquiring the vibration frequency of vocal cords is shown. An electronic device may include a radar, and the radar includes a transmitting antenna and a receiving antenna. During a call, the transmitting antenna may be used to transmit electromagnetic wave signals, and the receiving antenna may be used to receive returned electromagnetic wave signals.

[0075] In some scenarios, the transmitted electromagnetic wave signal may be referred to as a transmitted signal, and the returned electromagnetic wave signal may be referred to as echo data or a received signal. For ease of description, the following description will be given by taking the transmitted signal and the received signal as examples.

[0076] It is understandable that the received signal may include the reflected transmitted signal, and may also include the vibration frequencies of multiple parts of the user's body, such as the vibration frequency of the vocal cords, breathing, and chest when the user is talking. The received signal may also include the vibration frequency of the surrounding environment, such as the vibration frequency reflected by static objects such as walls. In some scenarios, the vibration frequency may also be called the vibration speed. For the sake of ease of description, the vibration frequency is used as an example in the following description.

[0077] Electronic devices can obtain data such as the vibration frequency of the user's vocal cords based on radar, and analyze the data to obtain the semantics that the user wants to express.

[0078] For example, Figure 4 A flowchart of a semantic recognition method according to an embodiment of the present application is shown.

[0079] S401. Turn on the radar function and select the mode.

[0080] In addition to being applied in call scenarios, the semantic recognition method of the embodiments of the present application can also be applied to live broadcast scenarios, silent call scenarios, and the like. Among them, the silent call scenario can be understood as a scenario in which the electronic device cannot obtain sound or the sound obtained is low, and content recognition cannot be performed through sound. Therefore, the interface of the electronic device may include different mode options such as call mode, live broadcast mode, and silent call mode. On the one hand, users can select the corresponding mode according to their respective needs. Based on the different modes selected by the user, the electronic device can execute different semantic recognition processes. On the other hand, the electronic device can also identify the current usage scenario. For example, the electronic device can identify that the current scenario is a live broadcast scenario or a call scenario, and the electronic device can execute different semantic recognition processes according to different usage scenarios.

[0081] For ease of description, a call scenario is used as an example for explanation below.

[0082] When using radar to recognize user semantics, the electronic device can provide the user with options related to radar functions. Figure 5 As shown, the interface 501 of the electronic device is a radar function interface, and the interface 501 may include options related to the radar function. For example, it may include real-time options, accuracy options, privacy options, and power saving mode options. The electronic device or the user may select the corresponding option according to the needs of the actual scenario.

[0083] Among them, the real-time options may include options such as fast, medium and slow. Exemplarily, when the real-time option is fast, it means that the current scene has a high demand for real-time, and the electronic device can recognize the user semantics based on the radar data as quickly as possible, but it may consume more traffic data, or the recognition accuracy is relatively low. When the real-time option is slow, it means that the current scene does not have a high demand for real-time, and the electronic device can accurately recognize the user semantics based on the radar data, but the recognition speed may be slow. When the real-time option is medium, the recognition effect of the electronic device on the radar data may be between the fast option and the slow option, which will not be repeated.

[0084] The accuracy options may include strict, accurate, and auxiliary options. Exemplarily, when the accuracy option is strict, it means that the recognition accuracy of call content based on radar data is relatively high, such as silent call scenarios, etc., then the electronic device can upload the data to the cloud server, so as to more accurately perform semantic recognition based on radar data. When the accuracy option is auxiliary, it means that the recognition accuracy of call content based on radar data is relatively low. The electronic device can mainly recognize the call content based on the voice signal received by the microphone, and semantic recognition based on radar data can be used as an auxiliary recognition method. At this time, the accuracy of semantic recognition based on radar data by the electronic device can be relatively low. When the real-time option is accurate, the recognition effect of the electronic device on radar data can be between the strict option and the auxiliary option, which will not be repeated.

[0085] Privacy options may include local processing and cloud processing. Cloud processing can be understood as uploading data to a cloud server for processing. Since cloud processing requires uploading radar data in electronic devices to a cloud server, data leakage may occur during data transmission. Therefore, when privacy requirements are high, radar data can be processed locally in the electronic device. When accurate identification of radar data is required and privacy requirements are not too high, radar data can be processed in the cloud.

[0086] The flow saving mode options may include options such as on and off. Since cloud processing requires uploading radar data in the electronic device to the cloud server, it consumes a certain amount of flow data. Therefore, when it is necessary to save flow data, radar data can be processed locally on the electronic device. When it is necessary to accurately identify radar data and there is no need to save flow data, radar data can be processed in the cloud.

[0087] It is understandable that the selection of each option in the interface 501 may conflict, for example, both real-time performance and accuracy are required. In a possible implementation, in order to reduce the conflict between options, after the user selects an option, the electronic device can set other options that conflict with the option to an unselectable state, such as graying them out, so that the user cannot select conflicting options at the same time, thereby improving the rationality of the electronic device's execution logic. Of course, the electronic device can also use other methods to reduce the conflict between options, which is not limited in the embodiments of the present application.

[0088] It is understandable that the interface 501 is an exemplary interface, and different electronic devices may have different radar function interfaces, and the interface 501 may include more or less content. The content included in the specific interface 501 is not limited in the embodiment of the present application.

[0089] S402: Radar data analysis.

[0090] The electronic device can analyze and process the radar data. In a possible implementation, the radar transmission signal can be in the form of a triangular wave pulse or a sawtooth wave pulse, etc., which is not limited in the present embodiment. Taking the sawtooth wave pulse as an example, the radar transmission signal S T (t) can satisfy the following formula:

[0091]

[0092] Among them, A T is the amplitude of the transmitted signal, j is the imaginary unit, f 0 is the starting frequency of the transmitted signal, t is the time of the transmitted signal, and S is the frequency modulation slope of the transmitted signal.

[0093] The corresponding phase of the transmitted signal is The following formula can be satisfied:

[0094]

[0095] The radar received signal S R (t) can satisfy the following formula:

[0096]

[0097] Among them, A R is the amplitude of the received signal, and τ is the time delay of the received signal relative to the transmitted signal.

[0098] It is understandable that, since the received signal is later than the transmitted signal, there may be a certain time delay between the transmitted signal and the received signal. The time delay τ may satisfy the following formula:

[0099]

[0100] Wherein, r is the radial distance of the target from the radar. In the embodiment of the present application, the target can be understood as the user. v is the vibration frequency of the vocal cords, which can also be understood as the radial motion speed of the target relative to the radar. c is the propagation speed of the electromagnetic wave, and the value can be taken as the speed of light c = 3×10 8 m / s.

[0101] The corresponding phase of the received signal is The following formula can be satisfied:

[0102]

[0103] It is understandable that, due to the loss of the transmission signal during the transmission process, the amplitude of the received signal is not necessarily equal to the amplitude of the transmitted signal, and there may be a certain phase difference between the transmitted signal and the received signal. In some scenarios, this phase difference can also be called the difference frequency phase. For the sake of ease of description, the difference frequency phase is used as an example for explanation.

[0104] Exemplarily, the difference frequency phase The following formula can be satisfied:

[0105]

[0106] Difference frequency signal S IF (t) can satisfy the following formula:

[0107]

[0108] Among them, A IF is the amplitude value of the difference frequency signal.

[0109] Taking the vibration frequency v of the vocal cords in the range of 100 Hz to 1000 Hz as an example, since the order of magnitude of the vibration frequency v of the human vocal cords and the speed of light c is quite different, after squaring the time delay τ, τ and τ 2 The order of magnitude difference between them is even greater, so the above formula can be simplified to S IF The last term in (t), then the difference frequency signal S IF (t) can satisfy the following formula:

[0110] s IF (t) = A IF exp(j2π(f o τ+Stτ)).

[0111] For the difference frequency signal S IF (t) Perform Fourier transform to obtain the difference frequency signal on the spectrum. The difference frequency signal f on the spectrum IF The following formula can be satisfied:

[0112]

[0113] Where λ is the wavelength of the electromagnetic wave.

[0114] Since the vibration frequency v of the vocal cords is of a large order of magnitude different from the speed of light c, the first term in the above formula can be simplified to obtain the difference frequency signal f on the following spectrum: IF :

[0115]

[0116] Furthermore, the radar receiving signal can be subjected to a fast Fourier transform (FFT) in the distance dimension to obtain the spatial distribution of the receiving signal, and then subjected to an FFT in the speed dimension to obtain the speed information of the receiving signal, which may include the vibration frequency of the vocal cords.

[0117] It is understandable that during a call, the user may have different vocal cord vibration frequencies for different text pronunciations. Therefore, the electronic device can analyze the semantics that the user wants to express based on the vocal cord vibration frequency.

[0118] For example, the vibration frequency of the vocal cords v i The following formula can be satisfied:

[0119]

[0120] Among them, T c is the duration of each sawtooth pulse, is the difference frequency phase of the vocal cords, which can also be understood as the difference frequency phase mentioned above

[0121] It can be understood that since the received signal may include the vibration frequencies of multiple parts of the user's body and the vibration frequencies of the surrounding environment, and in a call scenario, more attention is paid to the vibration frequency of the vocal cords, therefore, in an embodiment of the present application, the vibration frequency of the vocal cords can be called the target signal, and other vibration frequencies can be called the vibration frequencies of noise.

[0122] In the embodiment of the present application, the electronic device can process the received signal to enhance the target signal as much as possible, such as performing time-frequency focusing enhancement and other processing, and remove the vibration frequency of clutter in the received signal as much as possible, such as performing moving target display, constant false alarm detection, multipath suppression and other processing. In this way, the electronic device can more accurately parse the call semantics based on the target signal in the received signal, thereby improving the call quality.

[0123] S403: Display moving targets.

[0124] When the radar transmitting antenna sends a transmission signal to the surrounding space, the received signal may include the vibration frequency of the clutter reflected by the static objects in the surrounding space after being reflected by the static objects in the surrounding space. For example, in an indoor scene, due to the small space, the received signal may contain the vibration frequency of clutter of static objects such as indoor furniture, walls, ground, and ceiling. The vibration frequency of the clutter is mixed with the target signal, which may mask the target signal and interfere with the recognition of the target signal by the electronic device. Therefore, the embodiment of the present application can suppress the clutter of the received signal.

[0125] It is understandable that static objects hardly vibrate, so the vibration frequency of static objects can be zero or close to zero. Based on this, the embodiment of the present application can adopt a moving target cancellation method, using the difference in Doppler frequency between the target signal and the static object to filter out the zero-frequency component of the static object. In other words, the signal with a frequency of zero or close to zero can be filtered to highlight the target signal.

[0126] In a possible implementation, electronic devices can use recursive cancellers to describe the dynamic behavior and response of signals. For example, the output result of the previous moment is fed back to the input result of the next moment, and the clutter is suppressed by changing the feedback coefficient k, thereby achieving signal filtering and denoising.

[0127] Exemplarily, the output result at the previous moment and the input result at the next moment may satisfy the following differential equation:

[0128] y(n+1)=x(n+1)-x(n)+ky(n).

[0129] Wherein, n is the previous moment, n+1 is the next moment, x is the input result, y is the output result, and k is the feedback coefficient. It is understandable that k can be set by the electronic device according to an empirical value, for example, k can be a value between 0 and 1, and of course can also be other possible values. The specific value of k is not limited in the embodiment of the present application.

[0130] The above difference equation can be converted into an expression in the Z domain. That is, the variables in the difference equation are expressed as powers of Z, thus obtaining the following expression in the Z domain:

[0131] zY(z)=zX(z)-X(z)+kY(z).

[0132] Then the transfer function H(z) can satisfy the following formula:

[0133]

[0134] The corresponding frequency response can satisfy the following formula:

[0135]

[0136] Where T is the duration and ω is the frequency in the received signal.

[0137] It is understandable that the electronic device can multiply the received signal by the frequency response. When the frequency ω is 0, sinω is also 0, and the received signal is also 0. When the frequency ω is close to 0, sinω is also close to 0, and the received signal is also close to 0. Among them, ω can also take values ​​near the frequency period such as π, 2π, etc., which is not limited in the embodiments of the present application. In this way, the electronic device can filter the signal with a frequency of zero or close to zero in the received signal.

[0138] S404: Constant false alarm rate (CFAR) algorithm.

[0139] After performing the above step S403 to filter the vibration frequency of the static object included in the received signal, the electronic device may further suppress the clutter in the received signal through a constant false alarm detection algorithm, thereby more accurately identifying the target signal.

[0140] In a possible implementation, the CFAR algorithm can select the detection unit in the received signal based on experience or randomly. The vicinity of the detection unit can be called a protection unit range, and one or more reference unit ranges may be included outside the protection unit range. It can be understood that the size of the vicinity range can be set by the electronic device based on experience. For example, the vicinity range may include a range of 1 meter (m) around the detection unit. The specific size of the vicinity range is not limited in the embodiments of the present application. The size of the reference unit range can be set by the electronic device based on experience. For example, each reference unit range may include a range of 7 centimeters (cm). The specific size of the reference unit range is not limited in the embodiments of the present application.

[0141] It is understandable that the detection unit can be compared with the units within the reference unit range, but not with the units within the protection unit range.

[0142] In a call scenario, since the energy of the target signal is larger than the frequency energy of the surrounding environment, when the detection unit compares the frequency energy with the unit within the reference unit range, if the frequency energy of the detection unit is larger, it is more likely that the detection unit includes the target signal; if the frequency energy of the unit within the reference unit range is larger, it is more likely that the unit within the reference unit range includes the target signal.

[0143] After determining the target signal range, the part outside the target signal range can be suppressed, thereby achieving clutter suppression.

[0144] Exemplarily, the CFAR algorithm can select a sampling reference point from each reference unit range and sort the multiple sampling reference points according to the magnitude of the frequency energy. The CFAR algorithm can also multiply the sampling reference point and the scaling factor as the clutter detection threshold of the reference unit range where the sampling reference point is located. The clutter detection threshold varies randomly and can be adaptively adjusted according to the average power of its neighborhood.

[0145] Among them, the zoom factor can be set by the electronic device based on experience or scenarios, and the zoom factor may include a range of 0 to 1, for example, the zoom factor value is 0.5. Of course, the zoom factor may also include other values. The specific value of the zoom factor is not limited in the embodiments of the present application.

[0146] It is understandable that when the scaling factor is large, more signals are filtered out, and part of the target signal may be filtered out, resulting in reduced accuracy of semantic recognition. When the scaling factor is small, fewer signals are filtered out, and more clutter may be retained, resulting in poor clutter suppression. Therefore, the electronic device can set a more appropriate value for the scaling factor based on experience, which can filter out clutter and retain the target signal, thereby achieving better clutter suppression effect and higher accuracy of semantic recognition.

[0147] When the detection unit is compared with a certain reference unit range, if the frequency energy of the detection unit is greater than or equal to the clutter detection threshold of the reference unit range, the signal of the detection unit can be retained; if the frequency energy of the detection unit is less than the clutter detection threshold of the reference unit range, the signal of the detection unit can be suppressed.

[0148] Optionally, after sorting multiple sampling reference points according to the size of frequency energy, the sampling reference points corresponding to larger and / or smaller frequency energy in the sorting can be removed first, so that when calculating the clutter detection threshold, the influence of extreme data on the threshold value can be sorted, so that the clutter detection threshold can be set within a more reasonable range, so as to have a better detection effect in a clutter environment.

[0149] S405, multipath suppression.

[0150] During the transmission process, the radar's transmitted signal may be affected by the surrounding environment and may be reflected, refracted, scattered, etc., thereby generating one or more echo signals, which can be called the multipath effect. In some scenarios, the echo signal can also be called a multipath signal. For the sake of convenience, the following description will take the multipath signal as an example.

[0151] It can be understood that since the target signal includes relatively rich micro-motion feature information, and the multipath signal has micro-motion feature information similar to the target signal, the target signal and the multipath signal have a strong correlation, resulting in the multipath signal affecting the recognition of the micro-motion features of the target signal. Therefore, it is necessary to eliminate these multipath signals to further reduce the interference of the multipath signal on the target signal.

[0152] Among them, the micro-motion feature information can be understood as the feature information of the tiny movements or vibrations contained in the target signal. For example, in the fields of radar, communication, or sonar, the tiny movements of the target signal can be described by the micro-motion feature information. These micro-motion feature information may include the vibration frequency, phase change, or other motion-related features of the target, such as tiny shaking of the limbs or torso, chest ups and downs, throat vibrations, etc., which are not limited in the embodiments of the present application.

[0153] In a possible implementation, the electronic device may adopt a correlation coefficient-based multipath suppression method to calculate the correlation coefficient between the multipath signal and the target signal and determine the similarity between the multipath signal and the target signal.

[0154] like Figure 6 As shown, in each frame of the range doppler (RD) diagram, the electronic device can obtain the target signal and suspected multipath signals And calculate the correlation of the two signals. Among them, the signal close to the radar can be used as the target signal, and the signal far from the radar can be used as the suspected multipath signal. The target signal and the suspected multipath signal both include vectors of distance dimension and velocity dimension. In some scenarios, the RD graph can also be called the RD spectrum. For the convenience of description, the RD graph is used as an example for the following description.

[0155] The electronic device may perform normalization processing after taking the modulus of the vectors of the target signal and the suspected multipath signal, and calculate the correlation between the target signal and the suspected multipath signal to obtain the Pearson correlation coefficient.

[0156] Exemplarily, the Pearson correlation coefficient of the target signal and the suspected multipath signal may satisfy the following formula:

[0157]

[0158] in, is the covariance between the target signal and the suspected multipath signal, σ X is the standard deviation of the target signal, σ Y is the standard deviation of the suspected multipath signal.

[0159] Among them, the covariance can satisfy the following formula:

[0160]

[0161] Where E represents the expected value, μ x is the expected value of the target signal, μ Y is the expected value of the suspected multipath signal.

[0162] Based on the above formula, the correlation coefficient between the target signal and the suspected multipath signal can be obtained, and the correlation coefficient can satisfy the following formula:

[0163]

[0164] The electronic device can compare the correlation coefficient between the target signal and the suspected multipath signal with the correlation coefficient threshold, and determine whether the suspected multipath signal is a multipath signal. The correlation coefficient threshold can be set by the electronic device based on experience, for example, the correlation coefficient threshold can be set to 0.7, and the specific value of the correlation coefficient threshold is not limited in the embodiment of the present application.

[0165] If the correlation coefficient between the target signal and the suspected multipath signal is less than the correlation coefficient threshold, it means that the suspected multipath signal is not a multipath signal, and the suspected multipath signal will not be eliminated. If the correlation coefficient between the target signal and the suspected multipath signal is greater than or equal to the correlation coefficient threshold, it means that the suspected multipath signal is a multipath signal, and the electronic device can eliminate the multipath signal, thereby reducing the impact of the multipath signal on the target signal.

[0166] S406, time-frequency focusing enhancement.

[0167] Time frequency concentration (TFC) can be understood as the analysis window focusing on a certain signal frequency component at different times and frequencies in signal time-frequency analysis, which is used to describe the changing trend of this component with time and frequency.

[0168] It can be understood that, compared with clutter, the energy of the target signal is more concentrated near a certain frequency, which makes the target signal have more obvious time-frequency local characteristics and better time-frequency focusing; while clutter has more complex or scattered time-frequency characteristics due to interference from factors such as multipath effects, terrain conditions, and meteorological environment, and the clutter has poor focusing. Therefore, when performing time-frequency analysis on the received signal, the time-frequency focusing difference between the target signal and clutter can be used to adopt corresponding processing and optimization strategies, thereby improving the time-frequency focusing ability of the target signal and further enhancing the time-frequency extraction effect of the target signal.

[0169] In a possible implementation, when performing time-frequency analysis, Rayleigh entropy (rényi) can be used to judge the time-frequency focusing. For example, the smaller the Rayleigh entropy, the better the local energy focusing of the signal in time-frequency; the larger the Rayleigh entropy, the worse the local energy focusing of the signal in time-frequency.

[0170] For example, the Rayleigh entropy R α (x) can satisfy the following formula:

[0171]

[0172] Where x is a random variable, α is the exponent of Rayleigh entropy, and the value range of α can include [0,1] and (1,+∞), P i is the probability distribution of variable x, and n is the number of possible values ​​that variable x can take.

[0173] The embodiment of the present application may use Rayleigh entropy to describe a time frequency representation (TFR).

[0174]

[0175] TFR can be composed of multiple continuous frames and is used to describe the change of frequency over time. For each frame of RD diagram, the corresponding time-frequency information can be obtained, which is recorded as the frame-based time frequency representation (FTFR). Therefore, by calculating the Rayleigh entropy of FTFR, the energy distribution of the corresponding distance dimension of the current frame can be obtained. Furthermore, the synchronous compression function can be used to adjust the STFT processing result using the synchronous compression function, and the instantaneous frequency information ω 0 (t,ω) can satisfy the following formula:

[0176]

[0177] Where j is the imaginary unit, ω 0 (t,ω) is the instantaneous frequency in the time-frequency plane, and F(t,ω) is the time-frequency distribution coefficient. The energy of the time-frequency information TFR generated by STFT is distributed in [ω 0 -Δ,ω 0 +Δ], Δ can be understood as the frequency ω 0 A certain range around. When F(t,ω) is not equal to 0, ω 0 (t,ω) is equal to ω 0 , thereby concentrating the TFR energy. It can also be understood as 0 -Δ,ω 0 +Δ]'s ability is concentrated to ω 0 In addition, the ability to use target signals is enhanced.

[0178] It can be understood that after the above time-frequency focusing processing, the target signal has better local focusing in time and frequency, and the distinction between the target signal and clutter is more obvious, so that a clearer and more concentrated energy distribution of the target signal can be obtained.

[0179] S407: Adaptive time-frequency extraction.

[0180] In the process of identifying the vibration frequency of the vocal cords in the received signal, when the indoor space is large, the radar can obtain useful information within a certain range around the human body. For example, the radar can select the place with the strongest energy in the received signal as the target center point, and search around the target center point to obtain more complete useful information. In other words, adaptive time-frequency extraction can further extract the target signal in the received signal.

[0181] It is understandable that people cannot be regarded as point targets relative to radars. Different parts of the body have different radar cross sections (RCS). RCS can be understood as the scattering cross section of the target to the radar wave at a specific frequency, polarization and incident angle. Therefore, the received signal containing the target signal may span multiple range gates and change with the radial distance of the target from the radar. Among them, the range gate can be understood as the resolution of the radar, or it can be understood as the minimum distance that the radar can distinguish. For example, the resolution of the radar is 7cm, which can also be called a range gate of 7cm.

[0182] like Figure 7 As shown, the angle θ between the farthest point and the closest point of the human body from the radar can satisfy the following formula:

[0183]

[0184] Among them, H rader is the altitude of the radar, X min is the distance between the closest point of the human body and the radar.

[0185] The distance from the farthest point of the human body to the radar is X max The following formula can be satisfied:

[0186]

[0187] Then the farthest and nearest displacement difference ΔX can satisfy the following formula:

[0188] ΔX=H rader ·sinθ.

[0189] It can be seen from the above formula that the time-frequency extraction range is related to the radar height H rader, and the angle θ between the farthest point and the closest point of the human body from the radar. The farthest and closest displacement difference ΔX can also be understood as the time-frequency extraction range.

[0190] For example, assuming that the calculated time-frequency extraction range is 1m, when the radar determines that the distance to the center point of the detected target is 5m, the radar can also extract time-frequency information between 4m and 6m, so that a more complete target signal can be obtained.

[0191] The adaptive time-frequency extraction method of the embodiment of the present application can improve the time-frequency focusing of the signal by using synchronous compression transformation. The time-frequency focusing at the range gate containing the target signal is better than the time-frequency focusing at the range gate containing only clutter.

[0192] In a possible implementation, electronic equipment can use the time-frequency focusing indicator Rayleigh entropy to reflect the difference in time-frequency focusing of target signals and clutter, screen out the range gate containing the target signal, and extract time-frequency information.

[0193] Figure 8 A flow chart of time-frequency extraction is shown.

[0194] S801, obtaining the RD graph of current data.

[0195] The electronic device may convert the information acquired by the radar into an RD map and execute step S802.

[0196] S802, perform time-frequency transformation on different distance dimensions and synchronous compression transformation.

[0197] It should be noted that step S802 can be understood as the execution of the above-mentioned step S406. The electronic device can perform time-frequency focusing enhancement, calculate the range Doppler spectrum for each frame signal, and perform inverse Fourier transform to obtain the signal change information over time in the distance dimension. The electronic device can also use the short-time Fourier method to transform from the time domain to the frequency domain, thereby improving the aggregation of the target signal and increasing the time-frequency aggregation degree of the target signal and the clutter.

[0198] S803. Record the Rayleigh entropy of each dimension and sort them.

[0199] The electronic device can calculate the Rayleigh entropy at different positions in the space and sort the Rayleigh entropy. The Rayleigh entropy can be used to identify the concentration strength of the signal. It can be understood that the Rayleigh entropy of the target signal is less than the Rayleigh entropy of the clutter. The Rayleigh entropy can refer to the relevant description in the above step S406, which will not be repeated.

[0200] S804. Taking an average of some Rayleigh entropy data as a spatial noise background threshold.

[0201] The electronic device may also average some of the Rayleigh entropy data, for example, average the second half of the sorted Rayleigh array, as a spatial noise background threshold, which may also be called a Rayleigh entropy threshold.

[0202] If the Rayleigh entropy of a certain position in the space is less than the Rayleigh entropy threshold, it means that the position contains the target signal and needs to be retained. If the Rayleigh entropy of a certain position in the space is greater than or equal to the Rayleigh entropy threshold, it means that the position does not contain the target signal and can be filtered out.

[0203] S805, record the maximum energy distance dimension position Xmax in the received signal, and perform time-frequency transformation and synchronous compression transformation.

[0204] It can be understood that, in the RD diagram corresponding to the received signal, the position Xmax of the maximum energy can be regarded as the position of the center point of the target signal. The electronic device can search around based on the center point position to obtain a relatively complete target signal.

[0205] The electronic device can also perform time-frequency transformation and synchronous compression transformation on the RD diagram at the center point position to obtain a time-frequency diagram corresponding to the position, and the time-frequency diagram can include relevant information of the target signal.

[0206] S806. Update position X to Xmax+1 and search backward.

[0207] After processing the position Xmax, the electronic device can further search around the position Xmax. In this way, by gradually adding 1 to search around, the entire search range can be traversed, thereby obtaining a relatively complete target signal.

[0208] Optionally, the position X may also be updated to Xmax-1, that is, searching forward. The specific search direction is not limited in the embodiment of the present application.

[0209] S807: Is the position X smaller than the rear boundary of the motion range?

[0210] In order to avoid blind search, the electronic device can determine whether the position X is smaller than the rear boundary of the motion range. The rear boundary of the motion range can be set by the electronic device based on experience. For example, in a call scenario, the rear boundary of the motion range can be set to 10m, and of course it can also be set to other values. The specific range of the rear boundary of the motion range is not limited in the embodiment of the present application.

[0211] If the position X is smaller than the rear boundary of the motion range, it means that no place outside the search range has been searched yet, and the search can continue, and step S808 can be executed.

[0212] If the position X is greater than or equal to the rear boundary of the motion range, it means that the search has reached a place outside the search range and it is necessary to search in another direction, then step S812 can be executed.

[0213] S808: Perform time-frequency transformation and synchronous compression transformation on position X.

[0214] The electronic device may also perform time-frequency transformation and synchronous compression transformation on the RD diagram of position X to obtain a time-frequency diagram corresponding to position X, which may include relevant information of the target signal.

[0215] S809. Calculate the Rayleigh entropy.

[0216] It is understandable that if in the above step S803, after calculating the Rayleigh entropies of different positions in the space, the Rayleigh entropies of each position are also saved, then steps S808 and S809 may not be performed, but the Rayleigh entropy of the current position X is obtained from the saved Rayleigh entropies of each position. This can save the computing power of the electronic device and improve the efficiency of time-frequency extraction.

[0217] S810: Determine whether the Rayleigh entropy is less than the Rayleigh entropy threshold.

[0218] When the Rayleigh entropy of position X is less than the Rayleigh entropy threshold, it indicates that the position includes the target signal, and step S811 may be executed.

[0219] When the Rayleigh entropy of position X is greater than or equal to the Rayleigh entropy threshold, it indicates that the position does not include the target signal, and step S812 may be executed.

[0220] S811. Accumulate the current distance dimension time-frequency information, update the position X to X+1, and record the accumulated number.

[0221] The electronic device can accumulate and superimpose the time-frequency information corresponding to each position of the target signal, and record the number of accumulations, so as to continuously enrich the time-frequency information related to the target signal, facilitating the subsequent identification and analysis of the target signal.

[0222] S812, update position X to Xmax-1, and search forward.

[0223] After completing the search in a certain direction of the position Xmax, the electronic device can change direction and continue searching around the position Xmax, so that the entire search range can be traversed, thereby obtaining a relatively complete target signal.

[0224] It is understandable that if in step S806, the position X is updated to Xmax-1, then the current step position X can also be updated to Xmax+1, that is, searching backwards. The specific search direction is not limited in the embodiment of the present application.

[0225] S813: Is the position X greater than the front boundary of the motion range?

[0226] In order to avoid blind search, the electronic device can determine whether the position X is greater than the front boundary of the motion range. The front boundary of the motion range can be set by the electronic device based on experience. For example, in a call scenario, the front boundary of the motion range can be set to 10m, and of course it can also be set to other values. The front boundary of the motion range can be set to be equal to or different from the rear boundary of the motion range. The specific setting of the front boundary of the motion range is not limited in the embodiment of the present application.

[0227] If the position X is greater than the front boundary of the motion range, it means that no place outside the search range has been searched yet, and the search can continue, and step S814 can be executed.

[0228] If the position X is less than or equal to the front boundary of the motion range, it means that the search has reached a place outside the search range and it is necessary to search in another direction, then step S818 can be executed.

[0229] S814: Perform time-frequency transformation and synchronous compression transformation on position X.

[0230] The electronic device may also perform time-frequency transformation and synchronous compression transformation on the RD diagram of position X to obtain a time-frequency diagram corresponding to position X, which may include relevant information of the target signal.

[0231] S815. Calculate the Rayleigh entropy.

[0232] Similar to step S809, if in step S803, after calculating the Rayleigh entropies of different positions in the space, the Rayleigh entropies of each position are also saved, then step S814 and step S815 may not be performed, but the Rayleigh entropy of the current position X is obtained from the saved Rayleigh entropies of each position. This can save the computing power of the electronic device and improve the efficiency of time-frequency extraction.

[0233] S816. Determine whether the Rayleigh entropy is less than the Rayleigh entropy threshold.

[0234] When the Rayleigh entropy of position X is less than the Rayleigh entropy threshold, it indicates that position X includes the target signal, and step S817 may be executed.

[0235] When the Rayleigh entropy of position X is greater than or equal to the Rayleigh entropy threshold, it indicates that position X does not include the target signal. It is understandable that, since the signal around the human body is continuous, when position X does not include the target signal, it indicates that the boundary of the target signal has been reached, and there is no need to continue searching forward or backward, and step S818 can be executed.

[0236] S817, accumulate the current distance dimension time-frequency information, update the position X to X-1, and record the accumulated number.

[0237] The same as step S811, no further details are given.

[0238] S818. Perform normalization processing according to the accumulated number.

[0239] Since the number of time-frequency information accumulated at different times is different, it is impossible to uniformly and accurately reflect the target signal.

[0240] It can be understood that normalization can average the accumulated time-frequency information, so that after normalization, the time-frequency information can be reflected more accurately.

[0241] S819: Output the current frame time-frequency diagram.

[0242] After searching each position in the RD diagram, the electronic device can obtain full-dimensional information containing the target signal.

[0243] S408: Binarize the time Doppler matrix.

[0244] The electronic device can intercept the time-frequency diagram according to the vibration frequency range of the vocal cords to obtain the time-frequency diagram of the vocal cord vibration, and binarize the time-frequency diagram, which is convenient for the electronic device to subsequently identify the vocal cord vibration in different scenes.

[0245] After the binarization process, the electronic device can identify the vocal cord vibration according to different scenes.

[0246] If the current scene is a live broadcast, the electronic device can execute step S409 to determine the user's demand for real-time performance and execute different processes based on the real-time performance demand.

[0247] If the current scene is a call scene or a silent call scene, the electronic device can execute step S413 to determine the amount of data to be analyzed and perform different data processing based on the amount of data.

[0248] In addition, if it is a silent call scenario, since the electronic device cannot collect voice data, there is a higher demand for identifying the user's speaking content. The electronic device can also execute step S417 to obtain data collected by the camera and recognize the user's lip shape, thereby improving the accuracy of recognition.

[0249] It is understandable that the execution of step S413 and step S417 may not distinguish the order of precedence. Step S417 may be executed before step S413, for example, during the execution of step S401 to step S408, step S417 may be executed. Step S417 may also be executed after step S413, or may be executed simultaneously with step S413. The specific execution timing of step S417 is not limited in the embodiments of the present application.

[0250] S409: Whether the real-time requirement is high.

[0251] Electronic devices can determine the data processing process based on real-time requirements.

[0252] For example, when the real-time requirement is high, for example, in the above Figure 5 In the interface 501, when the user selects the real-time option as fast, the electronic device can execute step S410 to reduce the amount of data through principal component analysis, thereby speeding up the data processing speed and improving the real-time performance of recognizing the user's speech content.

[0253] When real-time requirements are low, such as in the above Figure 5 In the interface 501, when the user selects the real-time option as slow, the electronic device can execute step S415 to upload the data to the cloud server for processing.

[0254] Optionally, before executing step S415, step S414 may be executed first to perform principal component analysis on the data, which is not limited in the embodiment of the present application.

[0255] S410. Principal component analysis (PCA).

[0256] PCA is a commonly used data dimensionality reduction technique used to discover patterns and structures in data. For example, PCA can project the original data into a new coordinate system through linear transformation so that the projected data has the largest variance. In this way, the more important features in the data can be screened, thereby reducing the amount of data while retaining the information of the original data as much as possible. In other words, PCA can extract the characteristic data related to the user and filter out the features that are not related to the user.

[0257] Among them, the user-related characteristic data can be understood as relevant features for identifying semantics, such as the highest frequency of vocal cord vibration, the lowest frequency of vocal cord vibration, the ratio of the highest frequency to the lowest frequency and / or the duration of the vocal cord vibration frequency, etc.

[0258] After completing the principal component analysis, the electronic device may execute step S411 to perform local processing on the characteristic data related to the user.

[0259] S411, local processing.

[0260] Since principal component analysis can extract useful information from the received signal more quickly and reduce the amount of data to be processed, and local processing does not require additional time to upload to the cloud server, data processing locally on the electronic device can quickly complete the recognition of call content.

[0261] It is understandable that due to the limitations of local electronic devices, memory size, etc., the electronic device's ability to analyze and process data is smaller than that of the cloud server, resulting in the electronic device's accuracy in identifying call content being lower than that of the cloud server. Therefore, when real-time requirements are high, in order to ensure recognition speed, call content recognition can be performed locally on the electronic device, but some accuracy may be sacrificed.

[0262] In a possible implementation, in the above Figure 5 In the interface 501, when the user selects the accuracy option as auxiliary, it means that the user has a low requirement for the recognition accuracy of the call content, and the electronic device can perform data processing locally.

[0263] In a possible implementation, in the above Figure 5 In interface 501, when the user selects the privacy option as local processing, the electronic device can process data locally.

[0264] In a possible implementation, in the above Figure 5 In the interface 501, when the user selects the flow saving mode option to be turned on, the electronic device can process data locally. In this way, the electronic device does not need to upload data to the cloud server, saving the flow occupied by uploading data.

[0265] S412. Lightweight convolutional neural network.

[0266] Electronic devices can use lightweight convolutional neural networks to process the reduced-dimensional vocal cord vibration data to determine what the user is saying.

[0267] It can be understood that lightweight convolutional neural networks can be understood as convolutional neural network models for running in electronic devices or embedded systems, which can have fewer parameters and computational complexity to run in resource-constrained environments. These network models can adopt some specific techniques, such as depth-separable convolution, network pruning, and quantization, to reduce model size and computational requirements while maintaining high accuracy.

[0268] S413: Whether the data volume is greater than or equal to the threshold.

[0269] It is understandable that step S408 can obtain a relatively large matrix containing a large amount of data.

[0270] When the amount of data is greater than or equal to the threshold, it means that the amount of data contained in the matrix is ​​large, and the electronic device can execute step S414 to perform principal component analysis on the data. In this way, the amount of data uploaded to the cloud server can be reduced, traffic data can be saved, and the pressure on the cloud server to process data can be reduced.

[0271] When the amount of data is less than the threshold, it means that the amount of data contained in the matrix is ​​not too large, and the electronic device may not perform principal component analysis, but execute step S415 to upload the data to the cloud server for processing. In this way, the code execution efficiency can be improved and computing power can be saved.

[0272] Among them, the threshold can be set by the electronic device based on experience, and can also be set based on real-time requirements. The specific value of the threshold is not limited in the embodiments of the present application.

[0273] For example, when the real-time requirement is high, for example, in the above Figure 5 In the interface 501, when the user selects the real-time option as fast, the electronic device can set the threshold to a relatively small value, so that the probability that the data volume is greater than the threshold is increased, and step S414 can be executed to reduce the data volume through principal component analysis, thereby speeding up the data processing speed and improving the real-time recognition of the user's speech content.

[0274] When real-time requirements are low, such as in the above Figure 5 In interface 501, when the user selects the real-time option as slow, the electronic device can set the threshold to a relatively large value, so that the probability that the data volume is greater than the threshold is reduced, and step S415 can be executed to upload the data to the cloud server for processing.

[0275] S414. Principal component analysis.

[0276] The principal component analysis may refer to the relevant description of the above step S410 and will not be described in detail.

[0277] After the principal component analysis is completed, step S415 may be executed to upload the characteristic data related to the user to the cloud server for processing.

[0278] Optionally, after completing the principal component analysis, the data may not be uploaded to the cloud server, but may be processed locally on the electronic device, which is not limited in the embodiments of the present application.

[0279] S415, uploading to the cloud server for data processing.

[0280] Since cloud servers have strong computing power and / or strong storage capacity, cloud servers have strong data processing capabilities and can more accurately analyze and identify the user's speech content.

[0281] Since uploading to the cloud server takes a certain amount of time, the speed of uploading data to the cloud server for data processing is relatively slower than the speed of local data processing on the electronic device. When the real-time requirement is low, in order to ensure recognition accuracy, the call content can be uploaded to the cloud server for recognition, but some time may be sacrificed.

[0282] In a possible implementation, in the above Figure 5 In the interface 501, when the user selects the accuracy option as accurate, it means that the user has relatively high requirements for the recognition accuracy of the call content, and the electronic device can upload the data to the cloud server for processing.

[0283] In a possible implementation, in the above Figure 5 In interface 501, when the user selects the privacy option as cloud processing, the electronic device can upload data to the cloud server for processing.

[0284] In a possible implementation, in the above Figure 5 In the interface 501, when the user selects the option of saving data mode to be off, it means that the user does not pay much attention to the use of data traffic. Optionally, the electronic device can upload the data to the cloud server for processing. In this way, the user's speech content can be analyzed and recognized more accurately, thereby improving the user experience.

[0285] S416, convolutional neural network processing radar data.

[0286] The cloud server may include a convolutional neural network model obtained by analyzing radar data through big data. It is understandable that since the cloud server can have more computing resources and storage space, the cloud server can carry larger and more complex network models and can consider resource consumption less. This allows the convolutional neural network model of the cloud server to pay more attention to the accuracy and complexity of the model without being limited by device resources. Therefore, the convolutional neural network model based on the cloud server can analyze and process the radar data more accurately, thereby identifying the user's speech content.

[0287] S417: Obtain camera data.

[0288] In silent call scenarios, electronic devices can obtain camera data and perform lip recognition to improve the accuracy of user semantic recognition.

[0289] In a possible implementation, in the above Figure 5 In interface 501, when the user selects the accuracy option as strict, it means that the user has a high requirement for the recognition accuracy of the call content. The electronic device can combine the radar data and the data collected by the camera to conduct a comprehensive analysis of the call content in the cloud server, so as to obtain a more accurate recognition result.

[0290] S418, convolutional neural network processes camera data.

[0291] The cloud server may include a convolutional neural network model obtained by analyzing camera data through camera big data. The camera data may be processed based on the model to thereby recognize semantics based on the user's lip shape.

[0292] S419: Determine the recognition result.

[0293] When it is a silent call scenario, the cloud server can fuse the recognition results of the radar data and the camera data, and execute step S420 to output the recognition results of the call content.

[0294] S420: Output the call text subtitles or perform other processing.

[0295] The following specific embodiments are used to describe the method of the present application in detail. The following embodiments may be combined with each other or implemented independently, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0296] Fig. 9 The semantic recognition method of the embodiment of the present application is shown. The method includes:

[0297] S901. In the process of collecting audio and video data of the user, obtain the vibration frequency of the user's vocal cords based on radar.

[0298] In an embodiment of the present application, the process of collecting the user's audio and video data may include scenarios using a microphone and / or camera, such as a voice call scenario, a video call scenario, a silent call scenario, a live broadcast scenario, etc.

[0299] S902: Obtain a first sound recognition result for semantically recognizing the vibration frequency of the vocal cords.

[0300] In the embodiment of the present application, the process of obtaining the first sound recognition result can be as described above. Figure 4 The relevant description of step S412 or step S415 in the corresponding embodiment is not repeated here.

[0301] S903. Obtain a target sound recognition result of the audio and video data based on the first sound recognition result; the target sound recognition result is the first sound recognition result, or the target sound recognition result is obtained by processing the first sound recognition result and the second audio recognition result, or the target sound recognition result is obtained by processing the first sound recognition result, the second audio recognition result and the third image recognition result, wherein the second audio recognition result is obtained by semantically recognizing the audio in the audio and video data, and the third image recognition result is obtained by lip recognition of the user in the audio and video data.

[0302] In the embodiment of the present application, the target sound recognition result may include text obtained based on the first sound recognition result, for example, the text may include subtitles, etc.

[0303] It is understandable that the electronic device can obtain the target sound recognition result based only on the first sound recognition result, or can simultaneously consider the semantic recognition of the audio and / or the lip shape recognition of the user in the image, which is not limited in the embodiments of the present application.

[0304] For example, in a voice call scenario, since no camera is used to capture images, the electronic device can perform semantic recognition based only on the vibration frequency of the vocal cords, or it can perform semantic recognition by considering both the vibration frequency of the vocal cords and audio data.

[0305] In a silent call scenario, since the microphone cannot receive the user's voice, the electronic device can perform semantic recognition based only on the vibration frequency of the vocal cords, or it can simultaneously consider the vibration frequency of the vocal cords and the user's mouth shape in the image for semantic recognition.

[0306] In scenarios such as video calls and live broadcasts, since the vibration frequency of the vocal cords, audio information and / or image information can be obtained, the electronic device can obtain the target sound recognition result based only on the first sound recognition result, or it can simultaneously consider the semantic recognition of the audio and / or the lip shape recognition of the user in the image.

[0307] S904: Display the target voice recognition result.

[0308] The semantic recognition method provided in the embodiment of the present application can obtain data such as the vibration frequency of the user's vocal cords based on the radar of the electronic device, and analyze the data obtained by the radar to obtain the semantics that the user wants to express. The use of radar to detect vocal cord vibration has good privacy and transmission capabilities. In this way, the electronic device does not have to rely entirely on the processing of voice signals or lip shapes, so that even in noisy, dimly lit or obstructed environments, the user's semantics can be more accurately recognized, the call quality can be improved, and the user experience can be improved.

[0309] Optional, in Fig. 9 On the basis of the corresponding embodiment, before obtaining the first sound recognition result of semantic recognition of the vibration frequency of the vocal cords, it also includes: preprocessing the vibration frequency of the vocal cords in the distance dimension to obtain the time-frequency diagram corresponding to the vibration frequency of the vocal cords; and performing semantic recognition on the time-frequency diagram corresponding to the vibration frequency of the vocal cords.

[0310] In the embodiment of the present application, the preprocessing of the distance dimension of the vibration frequency of the vocal cords can refer to the above Figure 4 The description of step S407 of the corresponding embodiment is not repeated here.

[0311] The radar can search for the vibration frequency of the vocal cords in different distance dimensions and accumulate the searched information, thereby reducing the dimension of the data and obtaining a more complete vibration frequency of the vocal cords, which is convenient for further extraction of useful signals from the received signals.

[0312] Optional, in Fig. 9 On the basis of the corresponding embodiment, the radar is used to obtain an electromagnetic wave signal, which includes the vibration frequency of the vocal cords, and the vibration frequency of the vocal cords is preprocessed in the distance dimension, including: determining the first position of the maximum energy in the electromagnetic wave signal within the radar search range; searching for the second position in the direction of the boundary of the radar search range with the first position as the center; if the Rayleigh entropy of the second position is less than a preset value, retaining the signal corresponding to the second position; or, if the Rayleigh entropy of the second position is greater than or equal to the preset value, not retaining the signal corresponding to the second position; and outputting a time-frequency diagram based on the retained signal, the time-frequency diagram including the time-frequency diagram corresponding to the vibration frequency of the vocal cords.

[0313] In the embodiment of the present application, the first position can be understood as the above Figure 4 In the corresponding embodiment, the position Xmax of the maximum energy in the RD diagram corresponding to the received signal. The first position can be understood as the center point position of the vibration frequency of the vocal cords.

[0314] The boundary direction may include the boundary direction of each direction of the radar search range, which is not limited in the embodiment of the present application. Figure 4 In the corresponding embodiment, the direction of searching backward or forward.

[0315] The second position may include the position Xmax+1 in the backward search, or may include the position Xmax-1 in the forward search, etc., which is not limited in the implementation of the present application.

[0316] The preset value may include a value set by the electronic device based on experience, or may include a value obtained based on Rayleigh entropy. For example, the preset value may include the average value of the sum of Rayleigh entropies of each position or part of the position, etc., and the implementation of this application is not limited thereto.

[0317] If the Rayleigh entropy of the second position is less than the preset value, it means that the position includes the vibration frequency signal of the vocal cords, and the signal corresponding to the second position can be retained; if the Rayleigh entropy of the second position is greater than or equal to the preset value, it means that the position does not include the vibration frequency signal of the vocal cords, and the signal corresponding to the second position will not be retained.

[0318] It is understandable that the preprocessing of the distance dimension of the vibration frequency of the vocal cords can be specifically referred to the above Figure 8 The relevant description of the corresponding embodiments will not be repeated here.

[0319] The radar can select the place with the strongest energy in the received signal as the target center point, and search around the target center point, so as to obtain more complete and useful information, which is convenient for further extraction of the vibration frequency signal of the vocal cords in the received signal.

[0320] Optional, in Fig. 9 On the basis of the corresponding embodiment, before determining the first position of the maximum energy in the electromagnetic wave signal, it also includes: performing time-frequency transformation on different distance dimensions of the electromagnetic wave signal to obtain a time-frequency distribution diagram STFT of each distance dimension; determining a preset value based on the Rayleigh entropy of FTFR, and the preset value includes the average value of the sum of the Rayleigh entropies of part or all of the FTFR.

[0321] In the embodiment of the present application, the time-frequency transformation of different distance dimensions of the electromagnetic wave signal can refer to the above Figure 8 The relevant description of step S802 of the corresponding embodiment is not repeated here.

[0322] It can be understood that after the time-frequency transformation processing, the target signal has better local focusing in time and frequency, and the distinction between the target signal and clutter is more obvious, so that a clearer and more concentrated energy distribution of the target signal can be obtained.

[0323] Optional, in Fig. 9 On the basis of the corresponding embodiment, before the vibration frequency of the vocal cords is preprocessed in the distance dimension, it also includes: performing one or more of the following processing on the electromagnetic wave signal: moving target detection, constant false alarm detection, multipath suppression processing, and time-frequency focusing processing; wherein, moving target detection is used to filter signals in the electromagnetic wave signal with a frequency of zero or a frequency less than a preset frequency value, constant false alarm detection is used to filter signals in the electromagnetic wave signal with low frequency energy, multipath suppression processing is used to filter echo signals in the electromagnetic wave signal, the echo signal includes the signal reflected back when the radar's transmission signal encounters an obstacle, and the time-frequency focusing processing is used to enhance the time-frequency focusing capability of the vibration frequency of the vocal cords in the electromagnetic wave signal.

[0324] In the embodiment of the present application, moving target detection can refer to the above Figure 4 The relevant description of step S403 of the corresponding embodiment is not repeated here. Figure 4 The description of step S404 in the corresponding embodiment will not be repeated here. Figure 4 The relevant description of step S405 of the corresponding embodiment is not repeated here. The time-frequency focusing process can refer to the above Figure 4 The relevant description of step S406 of the corresponding embodiment is not repeated here.

[0325] It can be understood that the preset frequency value can be a value close to zero, or a value near a frequency period such as π, 2π, etc., which is not limited in the embodiment of the present application.

[0326] Moving target detection can filter out signals with a frequency of zero or near the frequency cycle in electromagnetic wave signals, thereby highlighting the target signal. Constant false alarm detection can filter out signals with low frequency energy in electromagnetic wave signals, suppress the clutter in the received signal, and thus more accurately identify the target signal. Multipath suppression processing can filter out echo signals in electromagnetic wave signals and reduce the interference of echo signals on target signals. Time-frequency focusing processing can utilize the time-frequency focusing difference between the target signal and clutter to enhance the time-frequency focusing ability of the vibration frequency of the vocal cords in the electromagnetic wave signal, thereby improving the time-frequency focusing ability of the target signal and further enhancing the time-frequency extraction effect of the target signal.

[0327] Optional, in Fig. 9 On the basis of the corresponding embodiment, before obtaining the first sound recognition result of semantic recognition of the vibration frequency of the vocal cords, it also includes: displaying a first interface, the first interface includes one or more of the following options: a first option, a second option, a third option, and a fourth option; wherein the first option is used to indicate the real-time requirement for semantic recognition of the vibration frequency of the vocal cords, and the first option includes a first level and a second level, and the real-time requirement of the first level is higher than that of the second level; the second option is used to indicate the accuracy requirement for semantic recognition of the vibration frequency of the vocal cords, and the second option includes a third level and a fourth level, and the accuracy requirement of the third level is higher than that of the fourth level; the third option is used to indicate the accuracy requirement for semantic recognition of the vibration frequency of the vocal cords The third option includes the fifth level and the sixth level, and the privacy requirement of the fifth level is higher than that of the sixth level; the fourth option is used to indicate the traffic consumption requirement for semantic recognition of the vibration frequency of the vocal cords, and the fourth option includes the seventh level and the eighth level, and the traffic consumption requirement of the seventh level is higher than that of the eighth level; when the first level, the fourth level, the fifth level or the seventh level is selected, the second level, the third level, the sixth level and / or the eighth level is not selectable; when the second level, the third level, the sixth level and / or the eighth level is selected, the first level, the fourth level, the fifth level and / or the seventh level is not selectable.

[0328] In the embodiment of the present application, the first interface may include the above Figure 5 For example, the first option may include a real-time option, the second option may include an accuracy option, the third option may include a privacy option, and the fourth option may include a flow saving mode option. The first interface may also include other options, which are not limited in the embodiments of the present application.

[0329] The first option may include multiple levels, such as fast, medium and slow in interface 501. When the first level is fast, the second level may be medium or slow. When the first level is medium, the second level may be slow.

[0330] The second option may include multiple levels, such as strict, accurate, and auxiliary in interface 501. When the third level is strict, the fourth level may be accurate or auxiliary, and when the third level is accurate, the fourth level may be auxiliary.

[0331] The third option may include multiple levels. For example, in interface 501, the fifth level is local processing and the sixth level is cloud processing.

[0332] The fourth option may include multiple levels. For example, in interface 501, the seventh level is on and the eighth level is off.

[0333] It is understandable that the selection of various options in the first interface may conflict, for example, both real-time performance and accuracy are required. In order to reduce the conflict between options, when the user selects an option, the electronic device can set other options that conflict with the option to an unselectable state, such as graying them out, so that the user cannot select conflicting options at the same time, thereby improving the rationality of the execution logic of the electronic device.

[0334] Optional, in Fig. 9 On the basis of the corresponding embodiments, when a selection operation of the first level, the fourth level, the fifth level or the seventh level is received, a first sound recognition result of semantically recognizing the vibration frequency of the vocal cords is obtained, including: using a convolutional neural network to perform semantic recognition on the vibration frequency of the vocal cords to obtain the first sound recognition result.

[0335] In an embodiment of the present application, when the user selects the accuracy option as auxiliary, it means that the user has lower requirements for the recognition accuracy of the call content, or when the user selects the privacy option as local processing, or when the user selects the data saving mode option as on, the electronic device can process data locally.

[0336] In this way, the electronic device does not need to upload data to the cloud server, saving the traffic occupied by uploading data. Since local processing does not require additional time to upload to the cloud server, data processing is performed locally on the electronic device, and the recognition of the call content can be completed faster.

[0337] Optional, in Fig. 9On the basis of the corresponding embodiments, when a selection operation for the second level, the third level, the sixth level and / or the eighth level is received, a first sound recognition result for semantically identifying the vibration frequency of the vocal cords is obtained, including: uploading the vibration frequency of the vocal cords to a cloud server, and obtaining the first sound recognition result for semantically identifying the vibration frequency of the vocal cords from the cloud server, the first sound recognition result being obtained by the cloud server using a convolutional neural network to perform semantic recognition on the vibration frequency of the vocal cords.

[0338] In the embodiment of the present application, when the user selects the accuracy option as accurate, it means that the user has relatively high requirements for the accuracy of identifying the call content, or when the user selects the privacy option as cloud processing, or when the user selects the data saving mode option as off, it means that the user is not very concerned about the use of data traffic, then the electronic device can upload the data to the cloud server for processing.

[0339] In this way, since the cloud server has strong computing power and / or strong storage capacity, the cloud server has a strong ability to process data, and can more accurately analyze and identify the user's speech content, thereby improving the user experience.

[0340] Optional, in Fig. 9 On the basis of the corresponding embodiment, before obtaining the first sound recognition result for semantic recognition of the vibration frequency of the vocal cords, it also includes: performing principal component analysis PCA on the vibration frequency of the vocal cords, extracting characteristic data related to the recognition semantics, the characteristic data including one or more of the following: the highest frequency of vocal cord vibration, the lowest frequency of vocal cord vibration, the ratio of the highest frequency to the lowest frequency, the duration of the vocal cord vibration frequency; obtaining the first sound recognition result for semantic recognition of the vibration frequency of the vocal cords includes: obtaining the first sound recognition result for semantic recognition of the characteristic data.

[0341] In the embodiment of the present application, the specific PCA can refer to the above Figure 4 The relevant description in step S410 of the corresponding embodiment is not repeated here.

[0342] PCA can filter out the more important features in the data, thereby reducing the amount of data while retaining the information of the original data as much as possible. In this way, the characteristic data related to the user can be extracted, and the features irrelevant to the user can be filtered out, so as to extract useful information from the received signal more quickly.

[0343] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0344] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0345] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0346] like Fig.10 FIG. 1 is a schematic diagram of a chip structure provided by an embodiment of the present application. The chip 1000 includes one or more (including two) processors 1001 , a communication line 1002 , a communication interface 1003 and a memory 1004 .

[0347] In some implementations, the memory 1004 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0348] The method described in the above embodiment of the present application can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 1001. The above processor 1001 can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1001 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0349] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1004, and the processor 1001 reads the information in the memory 1004 and completes the steps of the above method in combination with its hardware.

[0350] The processor 1001 , the memory 1004 , and the communication interface 1003 may communicate with each other via the communication line 1002 .

[0351] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0352] The embodiment of the present application also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0353] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0354] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0355] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A semantic recognition method, characterized in that: Applied to an electronic device, the electronic device includes a radar, and the method includes: In the process of collecting the audio and video data of the user, obtaining the vibration frequency of the vocal cords of the user based on the radar; Obtaining a first sound recognition result for semantically recognizing the vibration frequency of the vocal cords; A target sound recognition result of the audio and video data is obtained based on the first sound recognition result; the target sound recognition result is the first sound recognition result, or the target sound recognition result is obtained by processing the first sound recognition result and the second audio recognition result, or the target sound recognition result is obtained by processing the first sound recognition result, the second audio recognition result and the third image recognition result, wherein the second audio recognition result is obtained by semantically recognizing the audio in the audio and video data, and the third image recognition result is obtained by lip recognition of the user in the audio and video data; The target voice recognition result is displayed.

2. The method according to claim 1, characterized in that Before obtaining the first sound recognition result of semantically recognizing the vibration frequency of the vocal cords, the method further includes: Preprocessing the vibration frequency of the vocal cords in terms of distance dimension to obtain a time-frequency diagram corresponding to the vibration frequency of the vocal cords; Semantic recognition is performed on the time-frequency diagram corresponding to the vibration frequency of the vocal cords.

3. The method according to claim 2, characterized in that The radar is used to acquire an electromagnetic wave signal, wherein the electromagnetic wave signal includes the vibration frequency of the vocal cords, and preprocessing the vibration frequency of the vocal cords in a distance dimension includes: Determining a first position of maximum energy in the electromagnetic wave signal within the radar search range; Taking the first position as the center, searching for a second position in the direction of the boundary of the radar search range; If the Rayleigh entropy of the second position is less than a preset value, the signal corresponding to the second position is retained; or if the Rayleigh entropy of the second position is greater than or equal to the preset value, the signal corresponding to the second position is not retained; A time-frequency diagram is output based on the retained signal, wherein the time-frequency diagram includes a time-frequency diagram corresponding to the vibration frequency of the vocal cords.

4. The method according to claim 3, characterized in that Before determining the first position of the maximum energy in the electromagnetic wave signal, the method further includes: Performing time-frequency transformation on different distance dimensions of the electromagnetic wave signal to obtain a time-frequency distribution diagram STFT of each distance dimension; The preset value is determined based on the Rayleigh entropy of the FTFR, and the preset value includes an average value of the sum of the Rayleigh entropies of some or all of the FTFRs.

5. The method according to any one of claims 2 to 4, characterized in that: Before preprocessing the distance dimension of the vibration frequency of the vocal cords, the method further includes: Performing one or more of the following processing on the electromagnetic wave signal: moving target detection, constant false alarm detection, multipath suppression processing, and time-frequency focusing processing; Among them, the moving target detection is used to filter out signals in the electromagnetic wave signal whose frequency is zero or whose frequency is less than a preset frequency value, the constant false alarm detection is used to filter out signals in the electromagnetic wave signal with low frequency energy, the multipath suppression processing is used to filter out echo signals in the electromagnetic wave signal, and the echo signals include signals reflected back when the radar's transmission signal encounters an obstacle, and the time-frequency focusing processing is used to enhance the time-frequency focusing capability of the vibration frequency of the vocal cords in the electromagnetic wave signal.

6. The method according to any one of claims 1 to 5, characterized in that: Before obtaining the first sound recognition result of semantically recognizing the vibration frequency of the vocal cords, the method further includes: Displaying a first interface, wherein the first interface includes one or more of the following options: a first option, a second option, a third option, and a fourth option; The first option is used to indicate the real-time requirement for semantic recognition of the vibration frequency of the vocal cords, and the first option includes a first level and a second level, and the real-time requirement of the first level is higher than that of the second level; The second option is used to indicate the accuracy requirement for semantic recognition of the vibration frequency of the vocal cords, the second option includes a third level and a fourth level, and the accuracy requirement of the third level is higher than that of the fourth level; The third option is used to indicate the privacy requirement for semantically identifying the vibration frequency of the vocal cords, the third option includes a fifth level and a sixth level, and the privacy requirement of the fifth level is higher than that of the sixth level; The fourth option is used to indicate the traffic consumption requirement for semantic recognition of the vibration frequency of the vocal cords, and the fourth option includes a seventh level and an eighth level, and the traffic consumption requirement of the seventh level is higher than that of the eighth level; When the first level, the fourth level, the fifth level or the seventh level is selected, the second level, the third level, the sixth level and / or the eighth level is in an unselectable state; When the second level, the third level, the sixth level and / or the eighth level are selected, the first level, the fourth level, the fifth level and / or the seventh level are in an unselectable state.

7. The method according to claim 6, characterized in that In case of receiving a selection operation of the first level, the fourth level, the fifth level or the seventh level, obtaining a first sound recognition result of semantically recognizing the vibration frequency of the vocal cords includes: A convolutional neural network is used to perform semantic recognition on the vibration frequency of the vocal cords to obtain the first sound recognition result.

8. The method according to claim 6, characterized in that In case of receiving a selection operation on the second level, the third level, the sixth level and / or the eighth level, obtaining a first sound recognition result for semantically recognizing the vibration frequency of the vocal cords includes: The vibration frequency of the vocal cords is uploaded to a cloud server, and the first sound recognition result for semantically recognizing the vibration frequency of the vocal cords is obtained from the cloud server, wherein the first sound recognition result is obtained by the cloud server performing semantic recognition on the vibration frequency of the vocal cords using a convolutional neural network.

9. The method according to any one of claims 1 to 8, characterized in that: Before obtaining the first sound recognition result of semantically recognizing the vibration frequency of the vocal cords, the method further includes: Performing principal component analysis (PCA) on the vibration frequency of the vocal cords to extract characteristic data related to recognition semantics, wherein the characteristic data includes one or more of the following: the highest frequency of vocal cord vibration, the lowest frequency of vocal cord vibration, the ratio of the highest frequency to the lowest frequency, and the duration of the vocal cord vibration frequency; The obtaining of a first sound recognition result for semantically recognizing the vibration frequency of the vocal cords comprises: The first sound recognition result of semantically recognizing the characteristic data is obtained.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 9.

11. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 9.

Images