Audio signal coding method, decoding method, equipment and storage medium

Abstract

The embodiment of the invention provides an audio signal encoding method, an audio signal decoding method, equipment and a storage medium. The audio signal encoding method comprises the following steps: acquiring a first audio signal of a tth frame; determining a to-be-coded first target signal according to the target coding mode; encoding the first target signal to obtain a first encoded signal, and sending the first encoded signal to a second device; wherein the target coding mode comprises a first mode and a second mode; in the first mode, the first target signal is the first audio signal; and in the second mode, the first target signal is a second audio signal, the second audio signal is one of the first audio signal and a first residual signal with less information content, and the first residual signal is a residual between the first audio signal and a predicted signal of a t-th frame obtained by prediction. According to the method, the traditional coding and decoding methods are compatible, and the audio coding efficiency is further improved by introducing the neural network.

Description

Technical Field

[0001] The present application relates to the field of speech coding and decoding technology, and in particular to an audio signal coding method, decoding method, device and storage medium. Background Art

[0002] The encoding and decoding of audio signals (voice signals) are essential basic links in communication. With the development of communication technology and the popularization of digital devices, the demand for the transmission and storage of audio signals is growing. As a key technology for achieving efficient transmission and high-quality reproduction, audio signal encoding and decoding technology has become the basis of many fields such as communication, broadcasting, and storage. Traditional analog signal transmission methods have problems such as large bandwidth consumption, susceptibility to interference, and difficulty in long-distance transmission. In order to solve these problems, audio signal coding technology came into being. It converts the original analog audio signal into a digital signal through a compression algorithm and effectively compresses it to reduce the bandwidth and storage space required for transmission.

[0003] During the encoding process, the encoder analyzes the characteristics of the audio signal, such as frequency distribution, amplitude change, etc., and uses the corresponding algorithm to encode the audio signal. Common encoding algorithms include but are not limited to Linear Predictive Coding (LPC), transform coding, such as Moving Pictures Experts Group-Advanced Audio Coding (MPEG-AAC), Subband Coding (SBC), Adaptive Multi-Rate (AMR), G series audio codec algorithms, etc. These algorithms achieve effective data compression by removing redundant information from the audio signal. The decoding process is the reverse process of encoding. After the decoder receives the encoded audio data, it restores the audio signal close to the original sound quality according to the encoding algorithm. This process is crucial to ensure the clarity and intelligibility of voice communication.

[0004] However, existing coding and decoding technologies still face challenges in processing complex speech signals, especially how to further improve coding efficiency. For example, on extremely low-bandwidth networks, achieving higher compression rates while ensuring sound quality. Therefore, researching and developing more efficient and intelligent speech audio coding and decoding technologies to meet the growing communication and storage needs is an important direction for current technological development. Summary of the invention

[0005] At least one embodiment of the present application provides an audio signal encoding method, decoding method, device and storage medium to achieve efficient and intelligent audio encoding or decoding processing.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for encoding an audio signal, which is applied to a first device, including:

[0008] Acquire the first audio signal of the t-th frame;

[0009] Determining a first target signal to be encoded according to a target encoding mode;

[0010] Encode the first target signal to obtain a first coded signal, and send the first coded signal to the second device;

[0011] Among them, the target encoding mode includes a first mode and a second mode; in the first mode, the first target signal is the first audio signal; in the second mode, the first target signal is a second audio signal, the second audio signal is the one with less information between the first audio signal and the first residual signal, and the first residual signal is the residual between the first audio signal and the predicted signal of the tth frame.

[0012] Optionally, the above method further includes:

[0013] Decoding the first encoded signal to obtain a first decoded signal;

[0014] Enhance the signal to be enhanced to obtain a first enhanced signal;

[0015] Based on the first enhanced signal, predicting a prediction signal of the t+1th frame;

[0016] Among them, when the second audio signal is the first audio signal, the signal to be enhanced is the first decoded signal; when the second audio signal is the first residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame.

[0017] Optionally, when the target encoding mode is the second mode, the method further includes:

[0018] Sending first information, where the first information is used to indicate whether the first coded signal is obtained based on the residual signal coding.

[0019] Optionally, the step of enhancing the signal to be enhanced to obtain a first enhanced signal includes:

[0020] The signal to be enhanced is enhanced using a neural network model to obtain a first enhanced signal.

[0021] Optionally, the above method further includes:

[0022] Calculating a first encoding quality according to a difference between the first audio signal and the first enhanced signal;

[0023] Based on the first encoding quality, the neural network model is optimized.

[0024] Optionally, the above method further includes:

[0025] Determine the target coding mode;

[0026] Wherein, when the second device does not support decoding of the second audio signal, the target encoding mode is the first mode; when the second device supports decoding of the second audio signal, the target encoding mode is the second mode.

[0027] Optionally, the above method further includes:

[0028] Capability indication information of the second device is obtained, where the capability indication information is used to indicate whether the second device supports decoding of the second audio signal.

[0029] In a second aspect, an embodiment of the present application provides a method for decoding an audio signal, which is applied to a second device, including:

[0030] receiving a first coded signal and first information of a t-th frame, wherein the first information is used to indicate whether the first coded signal is obtained by encoding based on a residual signal;

[0031] Decoding the first encoded signal to obtain a first decoded signal;

[0032] The signal to be enhanced is enhanced to obtain a first enhanced signal of the t-th frame;

[0033] Among them, when the first information indicates that the first coded signal is encoded based on the residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame; when the first information indicates that the first coded signal is not encoded based on the residual signal, the signal to be enhanced is the first decoded signal.

[0034] Optionally, the above method further includes:

[0035] Based on the first enhanced signal, a prediction signal of the t+1th frame is predicted.

[0036] Optionally, the step of enhancing the signal to be enhanced to obtain a first enhanced signal of the t-th frame includes:

[0037] The signal to be enhanced is enhanced using a neural network model to obtain a first enhanced signal.

[0038] In a third aspect, an embodiment of the present application provides a first device, including:

[0039] A first acquisition module, used to acquire a first audio signal of the tth frame;

[0040] A first determination module, used to determine a first target signal to be encoded according to a target encoding mode;

[0041] An encoding module, used for encoding the first target signal to obtain a first encoded signal;

[0042] A sending module, configured to send the first coded signal to a second device;

[0043] Among them, the target encoding mode includes a first mode and a second mode; in the first mode, the first target signal is the first audio signal; in the second mode, the first target signal is a second audio signal, the second audio signal is the one with less information between the first audio signal and the first residual signal, and the first residual signal is the residual between the first audio signal and the predicted signal of the tth frame.

[0044] Optionally, the above device also includes:

[0045] A decoding module, used for decoding the first encoded signal to obtain a first decoded signal;

[0046] An enhancement module, used for enhancing the signal to be enhanced to obtain a first enhanced signal;

[0047] A prediction module, configured to predict a prediction signal of a t+1th frame based on the first enhanced signal;

[0048] Among them, when the second audio signal is the first audio signal, the signal to be enhanced is the first decoded signal; when the second audio signal is the first residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame.

[0049] Optionally, the sending module is further used to send first information when the target coding mode is the second mode, wherein the first information is used to indicate whether the first coded signal is obtained based on the residual signal coding.

[0050] Optionally, the enhancement module is further used to enhance the signal to be enhanced by using a neural network model to obtain a first enhanced signal.

[0051] Optionally, the above device also includes:

[0052] a calculation module, configured to calculate a first encoding quality according to a difference between the first audio signal and the first enhanced signal;

[0053] An optimization module is used to optimize the neural network model based on the first encoding quality.

[0054] Optionally, the above device also includes:

[0055] A second determination module is used to determine a target coding mode;

[0056] Wherein, when the second device does not support decoding of the second audio signal, the target encoding mode is the first mode; when the second device supports decoding of the second audio signal, the target encoding mode is the second mode.

[0057] Optionally, the above device also includes:

[0058] The second acquisition module is used to acquire capability indication information of the second device, where the capability indication information is used to indicate whether the second device supports decoding of the second audio signal.

[0059] In a fourth aspect, an embodiment of the present application provides a second device, including:

[0060] A receiving module, configured to receive a first coded signal and first information of a t-th frame, wherein the first information is used to indicate whether the first coded signal is obtained by coding based on a residual signal;

[0061] A decoding module, used for decoding the first encoded signal to obtain a first decoded signal;

[0062] An enhancement module, used for enhancing the signal to be enhanced to obtain a first enhanced signal of the t-th frame;

[0063] Among them, when the first information indicates that the first coded signal is encoded based on the residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame; when the first information indicates that the first coded signal is not encoded based on the residual signal, the signal to be enhanced is the first decoded signal.

[0064] Optionally, the above device also includes:

[0065] A prediction module is used to predict a prediction signal of the t+1th frame based on the first enhanced signal.

[0066] Optionally, the enhancement module is further used to enhance the signal to be enhanced by using a neural network model to obtain a first enhanced signal.

[0067] In a fifth aspect, an embodiment of the present application provides a first device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in the first aspect.

[0068] In a sixth aspect, an embodiment of the present application provides a second device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in the second aspect.

[0069] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the steps of the method described in any one of the first aspect or the second aspect are implemented.

[0070] In an eighth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in any one of the first aspect or the second aspect.

[0071] Compared with the prior art, the audio signal encoding method, decoding method, device and storage medium provided in the embodiments of the present application enable the first device to select a suitable encoding method according to the decoding capability of the communication counterpart to adapt to the decoding capability of the communication counterpart. While being compatible with traditional encoding and decoding methods, it also further improves the audio encoding efficiency by introducing neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0073] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0074] Figure 2 A schematic diagram of a basic method of conventional speech coding in the related art;

[0075] Figure 3 A schematic diagram of the basic method of speech coding after the introduction of neural network into related technologies;

[0076] Figure 4A schematic diagram of an audio signal encoding method and a decoding method according to an embodiment of the present application being applied to a first device and a second device;

[0077] Figure 5 A flow chart of an audio signal encoding method according to an embodiment of the present application;

[0078] Figure 6 A flowchart of a method for decoding an audio signal according to an embodiment of the present application;

[0079] Figure 7 This is a schematic diagram of the structure of a first device according to an embodiment of the present application;

[0080] Figure 8 This is a schematic structural diagram of a second device according to an embodiment of the present application.

[0081] Fig. 9 A schematic structural diagram of a first device according to another embodiment of the present application;

[0082] Fig.10 This is a schematic structural diagram of a second device according to another embodiment of the present application. DETAILED DESCRIPTION

[0083] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0084] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0085] It is worth noting that the technology described in the embodiments of the present application is not limited to the Long Term Evolution (LTE) / LTE-Advanced (LTE-A) system, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency Division Multiple Access (SC-FDMA) or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the described technology can be used for the above-mentioned systems and radio technologies as well as other systems and radio technologies. The following description describes a new radio (NR) system for example purposes, and NR terms are used in most of the following descriptions, but these technologies can also be applied to systems other than NR systems, such as 6th Generation (6G) communication systems.

[0086] Figure 1A block diagram of a wireless communication system applicable to an embodiment of the present application is shown. The wireless communication system includes a terminal 11 and a network side device 12. Among them, the terminal 11 can be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (Personal Digital Assistant, PDA), a handheld computer, a netbook, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR), a virtual reality (Virtual Reality, VR) device, a robot, a wearable device (Wearable Device), an aircraft (flight vehicle), a vehicle-mounted device (Vehicle User Equipment, VUE), a ship-mounted device, a pedestrian terminal (Pedestrian User Equipment, PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines or furniture, etc.), a game console, a personal computer (Personal Computer, PC), a teller machine or a self-service machine and other terminal side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application. The network side device 12 may include an access network device or a core network device, wherein the access network device may also be referred to as a radio access network (Radio Access Network, RAN) device, a radio access network function or a radio access network unit. The access network device may include a base station, a wireless local area network (Wireless Local Area Network, WLAN) access point (Access Point, AP) or a wireless fidelity (Wireless Fidelity, WiFi) node, etc.Among them, the base station may be referred to as a Node B (NB), an evolved Node B (eNB), a next generation Node B (gNB), a New Radio Node B (NR Node B), an access point, a Relay Base Station (RBS), a Serving Base Station (SBS), a Base Transceiver Station (BTS), a radio base station, a radio transceiver, a Basic Service Set (BSS), an Extended Service Set (ESS), a Home Node B (HNB), a Home Evolved Node B (home evolved Node B), a Transmission Reception Point (TRP) or other appropriate terms in the relevant field. As long as the same technical effect is achieved, the base station is not limited to specific technical vocabulary. It should be noted that in the embodiment of the present application, only the base station in the NR system is used as an example for introduction, and the specific type of the base station is not limited.

[0087] The core network equipment may include but is not limited to at least one of the following: core network node, core network function, mobility management entity (Mobility Management Entity, MME), access mobility management function (Access and Mobility Management Function, AMF), session management function (Session Management Function, SMF), user plane function (User Plane Function, UPF), policy control function (Policy Control Function, PCF), policy and charging rules function unit (Policy and Charging Rules Function, PCRF), edge application service discovery function (Edge Application Server Discovery Function, EASDF), unified data management (Unified Data Management, UDM), unified data storage (Unified Data Repository, UDR), home user server (Home Subscriber Server, HSS), centralized network configuration (CNC), network storage function (Network Repository Function, NRF), network exposure function (Network Exposure Function, NEF), local NEF (Local NEF, or L-NEF), binding support function (Binding Support Function, BSF), application function (Application Function, AF), etc. It should be noted that in the embodiments of the present application, only the core network device in the NR system is introduced as an example, and the specific type of the core network device is not limited.

[0088] like Figure 2 As shown in the figure, the basic methods of traditional speech coding in the related art can be divided into waveform coding and parameter coding. Waveform coding is to sample, quantize and encode the waveform signal of analog speech in the time domain to form a digital speech signal. Parameter coding is based on the pronunciation mechanism of human speech, finds the characteristic parameters that characterize speech, and encodes the characteristic parameters. At the same time, there are also coding methods that are a mixture of waveform coding and parameter coding.

[0089] In order to further improve the coding efficiency, speech and audio coding are gradually introduced into neural networks. The speech or audio signal is sent to the neural network / artificial intelligence (AI) for encoding, and the sound signal is restored through the neural network during decoding. The basic coding method is as follows Figure 3 shown.

[0090] The neural network-based encoding methods in related technologies usually have the following problems:

[0091] 1. The structure of the neural network is not fixed. Currently, each company or research institute has proposed its own neural network architecture, which is different in the number of network layers, the type of neural computing units in each layer, and the number of neural units in each layer.

[0092] 2. The parameters of the neural network are not fixed. Currently, various companies or research institutions are continuously training models, and the parameters of the same network will be updated quickly. Therefore, the encoder and decoder as well as the encoded voice and audio files are facing the problem of upgrading at any time.

[0093] 3. Traditional speech coding methods have achieved good results in speech and audio coding, and can achieve good coding efficiency for most scenarios, but they need to be improved in scenarios such as extremely low-rate coding. AI-based encoders are just getting started, and in the case of incompatibility with traditional encoders, further data accumulation is needed to achieve better coding performance.

[0094] To solve at least one of the above problems, the embodiments of the present application provide an audio signal encoding method and a decoding method, which can reduce or avoid the occurrence of the above situations, improve communication efficiency, and improve user experience.

[0095] The embodiment of the present application provides an audio signal encoding method and decoding method, which, while being compatible with traditional encoding and decoding methods, introduces a neural network to further improve the efficiency of audio encoding. In addition, when the decoding end does not support the neural network / neural network architecture, or the neural network parameters are different, the embodiment of the present application can also complete the parsing of the audio code stream and output a complete reconstructed sound signal.

[0096] Figure 4 1 is a schematic diagram of an audio signal encoding method and a decoding method of an embodiment of the present application applied to a first device and a second device, wherein the first device may be a terminal, and the second device may be a network side device, or the first device may be a network side device, and the second device may be a terminal. The terminal / network side device may serve as both an encoding end and a decoding end of an audio signal. For example, in an actual two-way communication scenario, a device may send an audio encoding signal to a peer device, and may also receive and decode an audio encoding signal sent by the peer device.

[0097] The method of the embodiment of the present application will be described below from the encoding end and the decoding end of audio coding. It is understandable that the device of the encoding end (such as a terminal or a network side device) may also have the relevant functions or modules of the decoding end. Similarly, the device of the decoding end (such as a terminal or a network side device) may also have the relevant functions or modules of the encoding end.

[0098] Figure 5 1 is a flow chart of the method for encoding an audio signal according to an embodiment of the present application when it is applied to a first device. The first device may be a terminal or a network side device, which is not specifically limited in the embodiment of the present application. Figure 5 As shown, the method includes:

[0099] Step 51: Acquire the first audio signal of the tth frame.

[0100] Here, the audio signal of the t-th frame (referred to as the first audio signal for ease of description) refers to the source audio signal of the t-th frame. The first device encodes the first audio signal and sends the encoded signal of the t-th frame (referred to as the first encoded signal for ease of description) to the second device. The second device is the peer device of the first device. The same is true for the source audio signals of other frames. Figure 5 Same approach.

[0101] Step 52: Determine a first target signal to be encoded according to a target encoding mode.

[0102] Here, the target coding mode includes a first mode and a second mode. In the first mode, the first target signal is the first audio signal; in the second mode, the first target signal is a second audio signal, the second audio signal is the first audio signal and the first residual signal, which has less information, and the first residual signal is the residual between the first audio signal and the predicted signal of the tth frame.

[0103] In this article, the first mode is sometimes referred to as the traditional coding mode, and the second mode is sometimes referred to as the AI ​​coding mode. In the first mode, the first audio signal can be directly encoded by a traditional coding algorithm, including but not limited to LPC, transform coding (such as MPEG-AAC), SBC, AMR, G series audio codecs and other algorithms. In the second mode, the audio signal with less information in the first audio signal and the first residual signal is encoded, and the coding algorithm used may also be the above-mentioned traditional coding algorithm. In addition, the prediction signal of the tth frame is obtained by predicting the audio signal of the t-1th frame based on the neural network enhancement. When t is equal to 1, the signal of the t-1th frame can take a zero signal. The neural network is pre-trained, and the training method will be described later. In addition, optionally, the neural network can be further trained by online training.

[0104] Step 53: encode the first target signal to obtain a first encoded signal, and send the first encoded signal to the second device.

[0105] Here, in the embodiment of the present application, the encoder may use the above-mentioned traditional encoding algorithm or other proprietary algorithms to encode the first target signal determined in step 52, thereby obtaining the encoded signal of the tth frame (i.e., the first encoded signal). The first encoded signal may be transmitted to the second device via wired or wireless means. For example, it may be transmitted to the second device via a communication network.

[0106] Through the above steps, the first device of the embodiment of the present application can choose to directly encode the first audio signal, or encode the one with less information in the first audio signal and the first residual signal, so that the first device can select a suitable encoding method according to the decoding capability of the communication peer (second device) to adapt to the decoding capability of the communication peer, while being compatible with traditional encoding and decoding methods, and further improve the audio encoding efficiency by introducing neural networks. For example, when the communication peer does not support neural networks / neural network architectures, or the neural network parameters are different, the first device can select the first mode; when the communication peer supports neural networks / neural network architectures, or the neural network parameters are the same, the first device can select the second mode. In this way, even if the communication peer does not support the decoding capability corresponding to the AI ​​encoding mode, it can ensure that the communication peer can complete the parsing of the audio code stream and obtain a complete reconstructed sound signal.

[0107] In addition, when the target coding mode is the second mode, the first device may also send first information to the second device when sending the first coded signal to the second device, where the first information is used to indicate whether the first coded signal is obtained based on the residual signal encoding.

[0108] In the embodiment of the present application, when the target encoding mode is the first mode or the second mode, the first device can reconstruct and enhance the first encoded signal to obtain a prediction signal of the t+1th frame.

[0109] Specifically, the first coded signal can be decoded by a decoder (such as a traditional decoder) to obtain a first decoded signal (i.e., an audio signal reconstructed by the decoder), and the decoding algorithm can be set according to the adopted coding algorithm. Then, the signal to be enhanced is enhanced to obtain a first enhanced signal, wherein, when the second audio signal is the first audio signal, the signal to be enhanced is the first decoded signal; when the second audio signal is the first residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal to the prediction signal of the t-th frame. For example, the signal to be enhanced is enhanced using a pre-trained neural network model to obtain a first enhanced signal of the t-th frame. Then, based on the first enhanced signal, a prediction signal of the t+1-th frame is predicted. When predicting the audio signal of the t+1-th frame, the embodiment of the present application can use various existing prediction algorithms to predict based on historical enhanced signals including the first enhanced signal of the t-th frame to obtain a prediction signal of the t+1-th frame. The embodiment of the present application does not limit the specific prediction algorithm here.

[0110] In an embodiment of the present application, the neural network model may be pre-trained. During the training process, the encoding quality may be calculated based on the difference between the enhanced signal of the i-th frame enhanced by the neural network model and the i-th frame audio signal of the source audio signal, and then the neural network model may be trained based on the encoding quality until a preset training end condition is met, thereby obtaining the neural network model.

[0111] Furthermore, the embodiment of the present application can also perform online training on the neural network model. For example, in the process of sending the first audio signal, the first device can calculate the first encoding quality according to the difference between the first audio signal and the first enhanced signal; then, based on the first encoding quality, the neural network model is optimized, thereby fine-tuning the neural network model.

[0112] In order to adapt to the decoding capability of the communication peer (second device), the first device can select a suitable encoding mode according to the decoding capability of the second device. Before the above step 52, the first device can further determine the target encoding mode, wherein, when the second device does not support decoding of the second audio signal, the target encoding mode is the first mode; when the second device supports decoding of the second audio signal, the target encoding mode is the second mode. It should be noted that here, whether the decoding of the second audio signal is supported is taken as an example to indicate the decoding capability of the second device. It can be understood that supporting the decoding of the second audio signal is to support the decoding of the code stream obtained by the AI ​​encoding mode above, that is, to support the decoding of the encoded signal obtained by encoding the original audio signal and the residual signal with less information. The residual signal is the residual between the original audio signal and the predicted audio signal.

[0113] In an embodiment of the present application, the first device may obtain capability indication information of the second device, and the capability indication information is used to indicate whether the second device supports decoding of the second audio signal. Furthermore, the first device determines the target encoding mode according to the capability indication information of the second device. The capability indication information of the second device may be pre-configured on the first device side, or may be obtained from the second device, or may be obtained from the third device side. For example, during the process of establishing a communication link with the second device, or after establishing a communication link, the first device sends capability request information to the second device, and the second device sends its own capability indication information to the first device according to the capability request information. Alternatively, both the first device and the second device send their own capability indication information to the third device, and the third device saves the capability indication information of each device. In this way, when the first device needs to communicate with the second device, it may request the capability information of the second device from the third device and receive the capability indication information of the second device sent by the third device.

[0114] Figure 6 The figure is a flow chart of the audio signal decoding method described in the embodiment of the present application when it is applied to a second device. The second device may be a network side device or a terminal, which is not specifically limited in the embodiment of the present application. Figure 6 As shown, the method includes:

[0115] Step 61: Receive a first coded signal and first information of the tth frame, where the first information is used to indicate whether the first coded signal is obtained by coding based on a residual signal.

[0116] Step 62: decode the first encoded signal to obtain a first decoded signal.

[0117] Here, according to the encoding algorithm on the first device side, the second device uses a corresponding decoding algorithm to perform decoding, thereby obtaining a first decoded signal.

[0118] Step 63: enhance the signal to be enhanced to obtain a first enhanced signal of the tth frame.

[0119] Here, when the first information indicates that the first coded signal is obtained by encoding based on the residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the t-th frame; when the first information indicates that the first coded signal is not obtained by encoding based on the residual signal, the signal to be enhanced is the first decoded signal. The prediction signal of the t-th frame is predicted by the second device based on historical enhanced signals including the enhanced signal of the t-1-th frame. The second device can use various existing prediction algorithms for prediction. The prediction algorithm used by the second device is the same as the prediction algorithm used by the first device. The embodiment of the present application does not limit the specific prediction algorithm here.

[0120] In addition, the second device can enhance the signal to be enhanced using a neural network model to obtain a first enhanced signal. Typically, the neural network model used by the second device is the same model as the neural network model used by the first device.

[0121] Through the above steps, the embodiment of the present application can realize decoding of the encoded signal sent by the first device.

[0122] Furthermore, the second device may also predict the prediction signal of the t+1th frame based on the first enhanced signal. For example, the second device predicts the prediction signal of the t+1th frame based on the historical enhanced signals including the first enhanced signal of the tth frame.

[0123] Next, combined with the previous Figure 4 The above method is further described by taking a more specific example of encoding and decoding of speech or audio signals. The frequency spectrum range of speech signals is usually narrower than that of audio signals.

[0124] The encoding process of the voice or audio signal on the first device side includes:

[0125] Step 1: Receive the voice or audio signal S at the current time t t , the received signal is sent to the mode selection module, the prediction error calculation module, and the coding quality calculation module at the same time.

[0126] Here, S in Step 1 t The signal duration can be 10ms, 20ms, or other durations.t The signal sampling rate can be 8K, 16K, 32K, and other sampling rates. t The signal bit depth can be 8bit, 16bit, or other bit depths. t The number of channels of a signal can be 1, 2, or any other number.

[0127] Step 2: The prediction error calculation module calculates the input speech or audio signal S at time t. t , the predicted speech or audio signal P at time t t , get the residual R of the speech or audio signal at time t t .

[0128] Step 3, the mode selection module first completes the selection of the traditional encoding mode and the AI ​​encoding mode. If it is the traditional encoding mode, the mode selection module inputs S t If it is AI encoding mode, the mode selection module calculates S t and R t The amount of information is calculated, and the output with the least amount of information is selected, which are recorded as AI original signal mode and AI residual mode respectively. There are many ways to calculate the amount of information, for example: one way is to calculate S t and R t The signal energy of the signal is selected, and the smaller energy is selected; one way is to calculate S t and R t The distribution on the spectrum is selected to have a narrow spectrum. This embodiment of the present application does not make any specific limitation on this.

[0129] Step 4: Encode the input signal in voice or audio format. The encoding method is one of the existing standards. The output is the encoded bit stream C. s . C s The code stream is completely compliant with the traditional encoding format.

[0130] Step 5, send Cs to the decoder for decoding. In the traditional encoding mode, C s The bitstream can be decoded on any standard decoder to obtain the complete bitstream; in AI encoding mode, C s The bitstream is decoded on any standard decoder to obtain a bitstream that is a mixture of the residual signal and / or the original signal (for example, the i-th frame is the residual signal, the i+1-th frame is the original signal, etc.).

[0131] Step 6: For the traditional coding mode or AI original signal mode, the decoded signal is recorded as SE t , sent to the neural network model for encoding enhancement, and the output is SO t .

[0132] Step 7, for AI residual mode, the decoded residual signal REt The residual summation module is sent to the predicted speech or audio signal P at time t. t Add and get PE t .

[0133] Step 8: Neural network model for PE t Do encoding enhancement and output as SO t .

[0134] Here, in Step 8, the neural network model can be a convolutional neural network or a LSTM, Transformer or other network, such as WaveNet or the like.

[0135] Step 9, the next frame prediction module, the input is the current reconstructed SO t , the output is the prediction signal P at the next moment, that is, at moment t+1 t+1 .

[0136] Step 10, the coding quality calculation module calculates the original signal S t， Rebuilt SO t For quality calculation, PESQ, POLQA, and other quality calculation algorithms can be used. The input of the coding quality calculation module is the speech or audio coding quality at the current time t, denoted as Q t .

[0137] Step 11, encoding quality Q t The error is input into the neural network training module as the error calculation value of the neural network training and used for training or optimization of the neural network model.

[0138] In Step 11, there can be multiple error calculation strategies for training the neural network model, such as cross entropy, minimum mean square error, etc.

[0139] The above steps are performed during the encoding process.

[0140] The decoding process of the audio signal on the second device side includes:

[0141] Step 1, send the received Cs to the decoder for decoding. In the traditional encoding mode, C s The bitstream can be decoded on any standard decoder to obtain the complete bitstream; in AI encoding mode, C s The bitstream is decoded on any standard decoder to obtain a mixed bitstream of the residual and the original.

[0142] If the decoder is a conventional decoder, the decoded signal SE t Output directly to complete the entire decoding process.

[0143] Step 2: If the decoder supports AI mode, the decoded signal SE is converted to traditional encoding mode or AI original signal mode. t , sent to the neural network model for encoding enhancement, and the output is SO t .

[0144] Step 3, for the AI ​​residual mode, the decoded residual signal RE t The residual summation module is sent to the predicted speech or audio signal P at time t. t Add and get PE t .

[0145] Step 4, PE t Do encoding enhancement and output as SO t . SO t That is the decoded output sound signal in AI mode.

[0146] Step 5, the next frame prediction module, the input is the current reconstructed SO t , the output is the prediction signal P at the next moment, that is, at moment t+1 t+1 .

[0147] The above steps are performed during the decoding process.

[0148] This example also provides Figure 4 A speech and audio coding device (i.e., the first device) shown includes a traditional codec, a mode selection module, a prediction error module, a next frame prediction module, a residual addition module, a coding quality calculation module, a neural network model, a traditional decoder, a coding quality-based optimization and network training strategy module, etc.

[0149] This example also provides Figure 4 A speech and audio decoding device (ie, the second device) shown includes a traditional decoder, a next frame prediction module, a residual addition module, a neural network model, etc.

[0150] As can be seen from the above examples, the embodiments of the present application introduce neural networks while being fully compatible with traditional encoding and decoding methods, further improving the efficiency of voice and audio encoding. When the decoding end does not support neural networks, or the neural network architecture and parameters are different, the parsing of voice or audio code streams can be completed, and a complete reconstructed sound signal can be output. The above scheme of the embodiments of the present application can be compatible with a large number of stock devices and can support a large amount of stock voice and audio content.

[0151] Based on the above scheme of the embodiment of the present application, efficient coding technology can be used to significantly reduce the network bandwidth required for audio data transmission between devices, thereby reducing the operating costs of network operators. For example, through advanced audio compression algorithms, data traffic can be reduced while maintaining sound quality, which means that mobile network operators can serve more users without increasing additional infrastructure investment. Secondly, with the rapid development of mobile Internet, users' demand for high-quality audio content is growing. High-quality voice and audio coding technology can provide a better user experience and enhance user stickiness, thereby improving user satisfaction and loyalty. This not only helps to attract new users, but also helps to retain existing users and increase user life cycle value. In addition, the advancement of coding technology also supports the development of new business models, such as online music, video streaming services, etc. Finally, with the promotion of 5G technology, voice and audio coding technology has broad application prospects in the fields of Internet of Things, autonomous driving, etc., and the above scheme of the embodiment of the present application can be widely used in the above scenarios.

[0152] Please refer to Figure 7 , the embodiment of the present application further provides a first device, including:

[0153] A first acquisition module 701 is used to acquire a first audio signal of a t-th frame;

[0154] A first determination module 702, configured to determine a first target signal to be encoded according to a target encoding mode;

[0155] The encoding module 703 is used to encode the first target signal to obtain a first encoded signal;

[0156] A sending module 704, configured to send the first coded signal to a second device;

[0157] Among them, the target encoding mode includes a first mode and a second mode; in the first mode, the first target signal is the first audio signal; in the second mode, the first target signal is a second audio signal, the second audio signal is the one with less information between the first audio signal and the first residual signal, and the first residual signal is the residual between the first audio signal and the predicted signal of the tth frame.

[0158] Through the above modules, the embodiments of the present application can achieve efficient and intelligent audio encoding or decoding processing.

[0159] Optionally, the above device also includes:

[0160] A decoding module, used for decoding the first encoded signal to obtain a first decoded signal;

[0161] An enhancement module, used for enhancing the signal to be enhanced to obtain a first enhanced signal;

[0162] A prediction module, configured to predict a prediction signal of a t+1th frame based on the first enhanced signal;

[0163] Among them, when the second audio signal is the first audio signal, the signal to be enhanced is the first decoded signal; when the second audio signal is the first residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame.

[0164] Optionally, the sending module is further used to send first information when the target coding mode is the second mode, wherein the first information is used to indicate whether the first coded signal is obtained based on the residual signal coding.

[0165] Optionally, the enhancement module is further used to enhance the signal to be enhanced by using a neural network model to obtain a first enhanced signal.

[0166] Optionally, the above device also includes:

[0167] a calculation module, configured to calculate a first encoding quality according to a difference between the first audio signal and the first enhanced signal;

[0168] An optimization module is used to optimize the neural network model based on the first encoding quality.

[0169] Optionally, the above device also includes:

[0170] A second determination module is used to determine a target coding mode;

[0171] Wherein, when the second device does not support decoding of the second audio signal, the target encoding mode is the first mode; when the second device supports decoding of the second audio signal, the target encoding mode is the second mode.

[0172] Optionally, the above device also includes:

[0173] The second acquisition module is used to acquire capability indication information of the second device, where the capability indication information is used to indicate whether the second device supports decoding of the second audio signal.

[0174] It should be noted that the device in this embodiment is a device corresponding to the method applied to the first device side, and the implementation methods in the above embodiments are all applicable to the embodiments of the device, and can also achieve the same technical effects. The above device provided in the embodiment of the present application can implement all the method steps implemented in the above method embodiment, and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment in this embodiment will not be specifically described here.

[0175] Please refer to Figure 8 , the embodiment of the present application further provides a second device, including:

[0176] A receiving module 801 is configured to receive a first coded signal and first information of a t-th frame, wherein the first information is used to indicate whether the first coded signal is obtained by coding based on a residual signal;

[0177] A decoding module 802, configured to decode the first encoded signal to obtain a first decoded signal;

[0178] An enhancement module 803 is used to enhance the signal to be enhanced to obtain a first enhanced signal of the t-th frame;

[0179] Among them, when the first information indicates that the first coded signal is encoded based on the residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame; when the first information indicates that the first coded signal is not encoded based on the residual signal, the signal to be enhanced is the first decoded signal.

[0180] Through the above modules, the embodiments of the present application can achieve efficient and intelligent audio encoding or decoding processing.

[0181] Optionally, the above device also includes:

[0182] A prediction module is used to predict a prediction signal of the t+1th frame based on the first enhanced signal.

[0183] Optionally, the enhancement module is further used to enhance the signal to be enhanced by using a neural network model to obtain a first enhanced signal.

[0184] It should be noted that the device in this embodiment is a device corresponding to the method applied to the second device side, and the implementation methods in the above embodiments are all applicable to the embodiments of the device, and can also achieve the same technical effects. The above device provided in the embodiment of the present application can implement all the method steps implemented in the above method embodiment, and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment in this embodiment will not be specifically described here.

[0185] A first device according to another embodiment of the present application, such as Fig. 9 As shown, it includes a transceiver 910, a processor 900, a memory 920, and a program or instruction stored in the memory 920 and executable on the processor 900; when the processor 900 executes the program or instruction, each process of the embodiment of the method for encoding an audio signal is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0186] The transceiver 910 is used to receive and send data under the control of the processor 900 .

[0187] Among them, Fig. 9 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 900 and various circuits of memory represented by memory 920 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 910 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 930 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0188] The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 can store data used by the processor 900 when performing operations.

[0189] Another embodiment of the present application provides a second device, such as Fig.10 As shown, it includes a transceiver 1010, a processor 1000, a memory 1020, and a program or instruction stored in the memory 1020 and executable on the processor 1000; when the processor 1000 executes the program or instruction, each process of the above-mentioned audio signal decoding method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0190] The transceiver 1010 is used to receive and send data under the control of the processor 1000 .

[0191] Among them, Fig.10In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1000 and memory represented by memory 1020. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1010 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 1000 is responsible for managing the bus architecture and general processing, and the memory 1020 may store data used by the processor 1000 when performing operations.

[0192] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned audio signal encoding method and decoding method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned audio signal encoding method and decoding method embodiments are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0194] It should be noted that, in this article, the terms "include", "include" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), including a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for encoding an audio signal, applied to a first device, characterized in that: include: Acquire the first audio signal of the t-th frame; Determining a first target signal to be encoded according to a target encoding mode; Encode the first target signal to obtain a first coded signal, and send the first coded signal to the second device; Among them, the target encoding mode includes a first mode and a second mode; in the first mode, the first target signal is the first audio signal; in the second mode, the first target signal is a second audio signal, the second audio signal is the one with less information between the first audio signal and the first residual signal, and the first residual signal is the residual between the first audio signal and the predicted signal of the tth frame.

2. The method according to claim 1, characterized in that Also includes: Decoding the first encoded signal to obtain a first decoded signal; Enhance the signal to be enhanced to obtain a first enhanced signal; Based on the first enhanced signal, predicting a prediction signal of the t+1th frame; Among them, when the second audio signal is the first audio signal, the signal to be enhanced is the first decoded signal; when the second audio signal is the first residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame.

3. The method according to claim 2, characterized in that When the target encoding mode is the second mode, the method further includes: Sending first information, where the first information is used to indicate whether the first coded signal is obtained based on the residual signal coding.

4. The method according to claim 2, characterized in that: The step of enhancing the signal to be enhanced to obtain a first enhanced signal includes: The signal to be enhanced is enhanced using a neural network model to obtain a first enhanced signal.

5. The method according to claim 4, characterized in that The method further comprises: Calculating a first encoding quality according to a difference between the first audio signal and the first enhanced signal; Based on the first encoding quality, the neural network model is optimized.

6. The method according to claim 1, characterized in that Also includes: Determine the target coding mode; Wherein, when the second device does not support decoding of the second audio signal, the target encoding mode is the first mode; when the second device supports decoding of the second audio signal, the target encoding mode is the second mode.

7. The method according to claim 6, characterized in that The method further comprises: Capability indication information of the second device is obtained, where the capability indication information is used to indicate whether the second device supports decoding of the second audio signal.

8. A method for decoding an audio signal, applied to a second device, characterized in that: include: receiving a first coded signal and first information of a t-th frame, wherein the first information is used to indicate whether the first coded signal is obtained by encoding based on a residual signal; Decoding the first encoded signal to obtain a first decoded signal; The signal to be enhanced is enhanced to obtain a first enhanced signal of the t-th frame; Among them, when the first information indicates that the first coded signal is encoded based on the residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame; when the first information indicates that the first coded signal is not encoded based on the residual signal, the signal to be enhanced is the first decoded signal.

9. The method according to claim 8, characterized in that Also includes: Based on the first enhanced signal, a prediction signal of the t+1th frame is predicted.

10. The method according to claim 8, characterized in that The step of enhancing the signal to be enhanced to obtain a first enhanced signal of the t-th frame includes: The signal to be enhanced is enhanced using a neural network model to obtain a first enhanced signal.

11. A first device, characterized in that: include: A first acquisition module, used to acquire a first audio signal of the tth frame; A first determination module, used to determine a first target signal to be encoded according to a target encoding mode; An encoding module, used for encoding the first target signal to obtain a first encoded signal; A sending module, configured to send the first coded signal to a second device; Among them, the target encoding mode includes a first mode and a second mode; in the first mode, the first target signal is the first audio signal; in the second mode, the first target signal is a second audio signal, the second audio signal is the one with less information between the first audio signal and the first residual signal, and the first residual signal is the residual between the first audio signal and the predicted signal of the tth frame.

12. The device according to claim 11, characterized in that Also includes: A decoding module, used for decoding the first encoded signal to obtain a first decoded signal; An enhancement module, used for enhancing the signal to be enhanced to obtain a first enhanced signal; A prediction module, configured to predict a prediction signal of a t+1th frame based on the first enhanced signal; Among them, when the second audio signal is the first audio signal, the signal to be enhanced is the first decoded signal; when the second audio signal is the first residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame.

13. The device according to claim 12, characterized in that The sending module is further used to send first information when the target coding mode is the second mode, where the first information is used to indicate whether the first coded signal is obtained based on the residual signal coding.

14. The device according to claim 12, characterized in that The enhancement module is further used to enhance the signal to be enhanced by using a neural network model to obtain a first enhanced signal.

15. The device according to claim 14, characterized in that Also includes: a calculation module, configured to calculate a first encoding quality according to a difference between the first audio signal and the first enhanced signal; An optimization module is used to optimize the neural network model based on the first encoding quality.

16. The device according to claim 11, characterized in that Also includes: A second determination module is used to determine a target coding mode; Wherein, when the second device does not support decoding of the second audio signal, the target encoding mode is the first mode; when the second device supports decoding of the second audio signal, the target encoding mode is the second mode.

17. The device according to claim 16, characterized in that Also includes: The second acquisition module is used to acquire capability indication information of the second device, where the capability indication information is used to indicate whether the second device supports decoding of the second audio signal.

18. A second device, characterized in that: include: A receiving module, configured to receive a first coded signal and first information of a t-th frame, wherein the first information is used to indicate whether the first coded signal is obtained by coding based on a residual signal; A decoding module, used for decoding the first encoded signal to obtain a first decoded signal; An enhancement module, used for enhancing the signal to be enhanced to obtain a first enhanced signal of the t-th frame; Among them, when the first information indicates that the first coded signal is encoded based on the residual signal, the signal to be enhanced is a first added signal, and the first added signal is obtained by adding the first decoded signal and the prediction signal of the tth frame; when the first information indicates that the first coded signal is not encoded based on the residual signal, the signal to be enhanced is the first decoded signal.

19. The device according to claim 18, characterized in that Also includes: A prediction module is used to predict a prediction signal of the t+1th frame based on the first enhanced signal.

20. The device according to claim 18, characterized in that The enhancement module is further used to enhance the signal to be enhanced by using a neural network model to obtain a first enhanced signal.

21. A first device, characterized in that: include: A transceiver, a processor, a memory, and a program or instruction stored on the memory and executable on the processor; When the processor executes the program or the instruction, the steps of the device according to any one of claims 1 to 7 are implemented.

22. A second device, characterized in that: include: A transceiver, a processor, a memory, and a program or instruction stored on the memory and executable on the processor; When the processor executes the program or the instruction, the steps of the device according to any one of claims 8 to 10 are implemented.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented, or the steps of the method according to any one of claims 8 to 10 are implemented.

24. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7, or implement the steps of the method according to any one of claims 8 to 10.

Images