Data processing method and apparatus

Abstract

Embodiments of the present application disclose a data processing method. The method is applied to a terminal device, and the method comprises: performing feature extraction on first data to be encoded, to obtain first feature data; using a first data encoding model indicated by first metadata to perform context prediction on the first feature data, to obtain encoding statistical information, wherein the first data encoding model is a data encoding model determined by means of consistency negotiation between the terminal device and a cloud server prior to encoding the first data, and the data quantization precision of the first data encoding model indicated by the first metadata matches the data quantization precision of a first data decoding model used by the cloud server; encoding the first feature data on the basis of the encoding statistical information to obtain a first bitstream; and compressing the first metadata and the first bitstream to obtain a compressed bitstream, wherein the compressed bitstream is used for being sent to the cloud server.

Description

A data processing method and device

[0001] The present application claims priority from the Chinese patent application No. 202410439035.9 filed on April 11, 2024, and entitled "A data processing method and device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of digital image processing, and in particular to a data processing method and device. BACKGROUND

[0003] In the field of digital image processing, a user can use a terminal device to send compressed image data to a cloud server. The cloud server receives the compressed image data, decodes the image data, and performs subsequent processing such as storage and sharing.

[0004] Specifically, the image data is encoded by an encoder in the terminal device, and correspondingly, the compressed code stream is decoded by a decoder in the cloud server. Due to the mismatch between the encoding and decoding of different devices, the data quality and transmission efficiency are affected. SUMMARY

[0005] The present application provides a data processing method and device for improving data quality and transmission efficiency.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a data processing method, which is applied to a terminal device and includes:

[0008] performing feature extraction on first data to be encoded to obtain first feature data;

[0009] performing context prediction on the first feature data using a first data encoding model indicated by first metadata to obtain encoding statistics, the first data encoding model being a data encoding model determined by the terminal device and a cloud server when performing consistency negotiation before encoding the first data, and the data quantization precision of the first data encoding model indicated by the first metadata matching the data quantization precision of a first data decoding model used by the cloud server;

[0010] encoding the first feature data according to the encoding statistics to obtain a first code stream;

[0011] compressing the first metadata and the first code stream to obtain a compressed code stream, the compressed code stream being used for sending to the cloud server.

[0012] In a first possible implementation manner of the first aspect, the method further includes:

[0013] The terminal device respectively acquires resource information of the terminal device and resource information of the cloud server;

[0014] According to the resource information of the terminal device and the resource information of the cloud server, consistency detection is performed on whether the terminal device and the cloud server respectively support quantization precision, and a consistency detection result is obtained;

[0015] According to the consistency detection result, the first data coding model is determined from the model set of the terminal device;

[0016] The first metadata is generated according to data quantization precision of the first data coding model.

[0017] In the first possible implementation manner of the first aspect, the first metadata is generated according to data quantization precision of the first data coding model, including:

[0018] According to data quantization precision of the first data coding model, an acceleration algorithm implementation manner of the first data coding model is determined;

[0019] The first metadata is generated according to data quantization precision of the first data coding model and the acceleration algorithm implementation manner of the first data coding model.

[0020] In the first possible implementation manner of the first aspect, the consistency detection is performed according to the resource information of the terminal device and the resource information of the cloud server on whether the terminal device and the cloud server respectively support quantization precision, including:

[0021] According to the resource information of the terminal device and the resource information of the cloud server, offline consistency detection is performed; or,

[0022] According to the resource information of the terminal device and the resource information of the cloud server, real-time interaction is performed with the cloud server to perform online consistency detection.

[0023] In a second aspect, the embodiments of the present application further provide a data processing method, the method being applied to a cloud server, and the method including:

[0024] Decoding compressed code stream from a terminal device to obtain decoded second feature data;

[0025] perform context prediction on the second feature data using a first data decoding model indicated by the second metadata, the first data decoding model being a data decoding model determined when the cloud server performs consistency negotiation before obtaining the compressed code stream, and a data quantization precision of the first data decoding model indicated by the second metadata matching a data quantization precision of a first data encoding model used by the terminal device;

[0026] decode third feature data from the compressed code stream according to the decoding statistical information;

[0027] reconstruct the third feature data according to a reconstruction model of the cloud server to obtain reconstructed data.

[0028] In a third aspect, the embodiments of the present application further provide a data processing apparatus, which can include a terminal device or a cloud server.

[0029] In the third aspect of the present application, the constituent modules of the data processing apparatus can also perform the steps described in the foregoing first aspect or second aspect and various possible implementation manners, for details, see the foregoing description in the first aspect or second aspect and various possible implementation manners.

[0030] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores instructions, when the instructions are run on a computer, cause the computer to perform the method in the foregoing first aspect or second aspect.

[0031] In a fifth aspect, the embodiments of the present application provide a computer program product containing instructions, when the instructions are run on a computer, cause the computer to perform the method in the foregoing first aspect or second aspect.

[0032] In a sixth aspect, the embodiments of the present application provide a communication apparatus, which can include a terminal device or a cloud server or a chip or the like entity, the communication apparatus includes a processor, a memory, the memory is used to store instructions, and the processor is used to execute the instructions in the memory, so that the communication apparatus performs the method in any one of the foregoing first aspect or second aspect.

[0033] In a seventh aspect, the present application provides a chip system, which includes a processor, used to support a terminal device or a cloud server to implement the functions involved in the foregoing aspects, for example, to send or process the data and / or information involved in the foregoing method. In a possible design, the chip system further includes a memory, the memory is used to save necessary program instructions and data of the terminal device or the cloud server. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0034] In an eighth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to perform the voice signal decoding method in the first aspect or any possible implementation manner of the first aspect.

[0035] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0036] In the embodiments of the present application, by adaptively selecting a data model with the best precision between the terminal device and the cloud server, the inconsistency problem of coding and decoding technology between different platforms is solved. Not only the information of the original data is preserved, but also the information loss or the appearance of abnormal data caused by the coding and decoding difference is avoided, thereby improving the data quality and transmission efficiency. In addition, in the embodiments of the present application, by using a more complex reconstruction method on the side of the cloud server with stronger computing resources, the embodiments of the present application realize targeted enhancement of the key region while keeping the semantic consistency, so that the quality of the key region is higher, and the embodiments of the present application can realize the semantic consistent asymmetric size model cooperation. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a schematic diagram of an architecture of a data center according to an embodiment of the present application;

[0038] FIG. 2a is a schematic diagram of an exemplary system according to an embodiment of the present application;

[0039] FIG. 2b is a schematic diagram of another exemplary system according to an embodiment of the present application;

[0040] FIG. 3 is a schematic diagram of an embodiment of a data processing method according to an embodiment of the present application;

[0041] FIG. 4 is a flowchart of a terminal communication scenario of end-cloud cooperation according to an embodiment of the present application;

[0042] FIG. 5 is a flowchart of a model preparation stage according to an embodiment of the present application;

[0043] FIG. 6 is a flowchart of size model cooperation according to an embodiment of the present application;

[0044] FIG. 7 is a network structure diagram of a scale parameter calculation module according to an embodiment of the present application;

[0045] FIG. 8 is a schematic diagram of a compression process according to an embodiment of the present application;

[0046] FIG. 9 is a schematic diagram of a mean value calculation module with compression according to an embodiment of the present application;

[0047] FIG. 10 is a schematic diagram of a decompression process according to an embodiment of the present application;

[0048] FIG. 11 is a schematic diagram of a mean calculation module according to an embodiment of the present application;

[0049] FIG. 12a is a schematic diagram of an image enhancement framework according to an embodiment of the present application;

[0050] FIG. 12b is a schematic diagram of an image enhancement module according to an embodiment of the present application;

[0051] FIG. 12c is a schematic diagram of an image enhancement module according to an embodiment of the present application;

[0052] FIG. 13 is a schematic diagram of a feature classifier model according to an embodiment of the present application;

[0053] FIG. 14a is a schematic diagram of a decoding matching result according to an embodiment of the present application;

[0054] FIG. 14b is a schematic diagram of a decoding matching result according to an embodiment of the present application;

[0055] FIG. 14c is a schematic diagram of a decoding matching result according to an embodiment of the present application;

[0056] FIG. 14d is a schematic diagram of a decoding matching result according to an embodiment of the present application;

[0057] FIG. 15 is a schematic diagram of a terminal device according to an embodiment of the present application;

[0058] FIG. 16 is a schematic diagram of a cloud server according to an embodiment of the present application;

[0059] FIG. 17 is a schematic diagram of a computing device according to an embodiment of the present application;

[0060] FIG. 18 is a schematic diagram of a computing device cluster according to an embodiment of the present application;

[0061] FIG. 19 is a schematic diagram of a computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The terms used in the implementation part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0063] Those of ordinary skill in the art can know that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0064] In the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a way of distinguishing the objects with the same properties used in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or equipment including a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0065] At present, in the field of digital image processing, users can use terminal devices to transmit image data across devices, for example, users can compress image data through image encoding, thereby reducing the required storage space and transmission bandwidth of image data, and after obtaining the compressed code stream, the user can transmit the compressed code stream to the cloud server. After receiving the compressed code stream, the cloud server stores, shares or subsequent processes it.

[0066] The traditional codec can be, for example, JPEG, which is a format of lossy encoding of pictures. In the codec process of JPEG, Discrete Cosine Transform (DCT) and inverse DCT transform from color space to frequency domain space are involved, and floating-point numbers can be used for calculation, and then quantization operation is used to approximate to integers. When decoding in different devices, the floating-point calculation in the inverse DCT transform may cause differences in the calculation results, the calculation dimension is low, for example, at most 8 numbers are summed, and the cumulative error is small, and the slight error in the color space is not easy to perceive visually, so the calculation consistency problem has not attracted attention.

[0067] Traditional codec has low coding efficiency, and the reconstructed quality of pictures is not as good as artificial intelligence (AI).

[0068] In the prior art, compared with the traditional coding method, the AI coding technology has higher coding efficiency and better reconstructed quality. In the terminal device, the image data is encoded by an AI encoder, and correspondingly, in the cloud server, the compressed code stream can be decoded by an AI decoder. Due to the differences in computing accuracy and performance between different devices, the AI coding may not match, thereby affecting the image quality and transmission efficiency.

[0069] It can be understood that the technology of using deep learning to code and decode data is collectively referred to as AI coding.

[0070] In the embodiments of the present application, the terminal device side is referred to as "terminal side" or "mobile phone side", and the cloud server side is referred to as "cloud side" or "cloud side". The terminal side and the cloud side can be collectively referred to as "terminal cloud side" or "terminal cloud". For the AI encoder and AI decoder of the terminal cloud cooperation, the main difficulties in solving the above problems are:

[0071] 1. Cross-platform coding consistency. The computing accuracy of different devices may be different, for example, the floating point computing accuracy of the mobile phone side and the cloud server side is different. This may cause the encoder and decoder to produce inconsistent statistical results on different devices, thereby affecting the consistency of the terminal cloud.

[0072] 2. End-cloud size model semantic consistency. The computing resources of the mobile phone side and the cloud server side are different. The mobile phone side is usually limited by computing power and memory, while the cloud server has more powerful resources.

[0073] How to maximize the use of the difference between the terminal cloud resources, while ensuring that the image can be correctly decoded, improve the quality of image recovery, so that the cloud server side generates pictures with the same semantic consistency as the mobile phone side and better visual quality.

[0074] The method of the embodiments of the present application can be applied to a computing device cluster which can include one or more computing devices.

[0075] Wherein, the type of any computing device is not limited here. Exemplarily, any computing device can be a terminal device, a cloud server, a container or a virtual machine, etc.

[0076] The computing device cluster can provide a data processing platform through which the data processing method of the embodiments of the present application is implemented. Specifically, the data processing platform can be provided in a terminal device and in a cloud server, and the consistency and calculation accuracy of the terminal side and the cloud side can be achieved based on the respective data processing platforms of the terminal device and the cloud server, while the differences in computing power of different devices can also be fully utilized. The specific form of the computing device cluster and the corresponding data processing platform is not limited here.

[0077] In an example, the computing device cluster can be used to implement a cloud management platform, in other words, the data processing platform of the embodiments of the present application can be implemented through the cloud management platform.

[0078] The cloud management platform is used to manage the infrastructure that provides cloud services, and can provide computing, network and storage capabilities based on services of hardware resources and software resources. For example, the cloud management platform can include one or more data centers to provide cloud resources through the one or more data centers.

[0079] The data center will be introduced below in conjunction with an architecture schematic diagram shown in FIG. 1.

[0080] In FIG. 1, in the data center, the cloud management platform interacts with one or more servers (e.g., server 1 and server 2 in FIG. 1) through an internal network of the data center. The server includes a hardware layer and a software layer. The hardware layer includes hardware configured for the server, and the software layer includes an operating system installed and running on the server (the operating system of the virtual machine can be referred to as a host operating system). The host operating system is provided with a virtual machine manager (also referred to as a hypervisor), which is configured to implement computing virtualization, network virtualization, and storage virtualization of the virtual machine, and is responsible for managing the virtual machine. The virtual machine refers to a complete computer system that is simulated by software and runs in a completely isolated environment. In the system architecture shown in FIG. 1, a plurality of servers are provided in a data center, and the servers can be used to run virtual machines. The specifications of the virtual machines can be the same or different. The virtual machine can also be referred to as a cloud server (ECS), an elastic instance, etc. Different cloud service providers can have different names.

[0081] In an example of an embodiment of the present application, the cloud management platform can be a public cloud platform. At this time, a cloud service provider such as an individual or a software developer who has a cloud resource development capability can provide cloud services for users, and the users can obtain the cloud services through the Internet, but do not own the cloud computing resources.

[0082] Specifically, in the example shown in FIG. 1, the cloud management platform can provide an access interface (e.g., an interface or an API). A user of the cloud management platform and a cloud service provider can operate a client to remotely access the access interface to register a cloud account and a password in the cloud management platform, and after the cloud account and the password are successfully authenticated in the cloud management platform, the user logs in to the cloud management platform, thereby creating, managing, logging in to, and operating a virtual machine in the cloud data center, and executing a corresponding task through a cloud resource of the cloud management platform.

[0083] For example, as shown in the example of FIG. 2a, the data processing platform implementing the data processing method in the embodiments of the present application can be provided to the user in the form of a cloud service.

[0084] When a data processing task requiring graph data cross-device storage needs to be performed, some enterprises, organizations or individuals can purchase a cloud service including the data processing platform, so as to perform the related data processing task through the cloud resources of the cloud management platform and obtain the processing result of the input data from the cloud management platform.

[0085] The data processing platform can be implemented based on one or more cloud data centers and the like in the cloud management platform.

[0086] The data processing platform can be provided to the user as a separate cloud service, and can also be compatible with TensorFlow, PyTorch and the like, that is, can be used as an operator in TensorFlow, PyTorch and the like, to provide functions such as preprocessing, calculation, feature extraction and / or other data processing of real-time data and offline data in the computing framework.

[0087] Of course, the cloud management platform can also be other types of cloud management platforms, which are not limited in the embodiments of the present application.

[0088] In another example, as shown in FIG. 2b, the data processing platform can be provided to the user in the form of a software product or a function service in the software product. The software product can be deployed in the terminal device of the user or the server cluster of the enterprise user. The user can purchase a software license to obtain the use right of the software product, and can implement the data processing method in the embodiments of the present application through the software product within the license period.

[0089] The following takes the image data field as an example to exemplarily introduce the actual application mode of the data processing platform implementing the data processing method.

[0090] In an exemplary application scenario, the user can want to transmit the photos taken on the mobile phone to the cloud server for storage, sharing or subsequent processing. The user encodes the image data by using an AI model through TensorFlow, PyTorch and the like, and then sends the encoded code stream through the transmission network between the mobile phone and the cloud server. The computing framework can be deployed in the client device, and the cloud server can also deploy the computing framework. After receiving the compressed image data, the decoding processing is provided through the cloud resources of the cloud management platform.

[0091] It can be understood that the above example is only an exemplary introduction of an application scenario, and the data processing method in the embodiments of the present application can be applied to multiple fields, which are not limited herein.

[0092] In the embodiment of the present application, when the data processing platform is used to implement the data processing method in the embodiment of the present application, the encoding of image data can be realized, and the encoding of voice data and video data can also be applied. The above data processing can be the processing of stream data in a real-time scene, and the processing of batch data in an offline scene can also be realized.

[0093] In the embodiment of the present application, before the terminal device encodes data, the terminal device can perform model preparation with the cloud server. In the model preparation stage, the terminal device and the cloud server can negotiate model consistency, so that the terminal device can determine the data encoding model and the data decoding model that the terminal device can use, and the cloud server can also determine the data decoding model that the terminal device can use. Therefore, through the model preparation stage, the terminal device and the cloud server can realize consistency of function calculation across platforms.

[0094] In the embodiment of the present application, the terminal device and the cloud server negotiate model consistency, so that the data encoding model of the terminal device matches the data decoding model that the cloud server can use. The matching here can be that the data encoding model and the data decoding model have the same quantization calculation precision, or when the data encoding model and the data decoding model have the same quantization calculation precision, they also have the same acceleration algorithm implementation. The acceleration algorithm implementation is hereinafter referred to as "implementation". For example, models with different calculation precisions both provide two implementations: one is to enable acceleration algorithm optimization (such as enabling winograd acceleration algorithm), and the other is to disable acceleration algorithm optimization. The data encoding model of the terminal device and the data decoding model of the cloud server use the same implementation, for example, both disable acceleration algorithm optimization or both enable acceleration algorithm optimization.

[0095] In the embodiment of the present application, after the terminal device determines the quantization calculation precision and the implementation of the data encoding model, the terminal device can write the quantization calculation precision and the implementation of the data encoding model into the metadata. Similarly, after the cloud server determines the quantization calculation precision and the implementation of the data decoding model, the cloud server also writes the quantization calculation precision and the implementation of the data decoding model into the metadata.

[0096] In the embodiment of the present application, the data encoding model of the terminal device matches the data decoding model that the cloud server can use, but the image reconstruction model of the terminal device and the image reconstruction model of the cloud server can be determined according to the computing power of each device, that is, the terminal device and the cloud server can use different reconstruction algorithms. By taking advantage of the difference in computing power of different devices, the cloud server has stronger computing power and can perform more complex image reconstruction operations. This difference should be maximized to improve the efficiency of image recovery and enhancement while ensuring correct decoding of images.

[0097] In the embodiments of the present application, the terminal device can select a data coding model from a model set. The terminal device and the cloud server can use different model sets. The AI frameworks used by the terminal device and the cloud server are different, so the model formats are different, but the model accuracy and implementation methods in the model sets of the terminal device and the cloud server are the same. For example, the model set can be a model pool.

[0098] In the embodiments of the present application, after the terminal device and the cloud server each complete model preparation, the terminal device can determine a preconfigured data coding model, and the cloud server can also determine a preconfigured data decoding model. Then the terminal device can perform data reading, then call the pre-determined data coding model for coding, and finally send the compressed code stream to the cloud server. The compressed code stream can include compressed information and metadata on the terminal device side.

[0099] In some embodiments of the present application, after the terminal device generates the compressed code stream, the terminal device can decode through the data decoding model of the terminal device, and then reconstruct using the end-side reconstruction model of the terminal device to obtain an end-side reconstructed image. Because the computing power of the terminal device is different from that of the cloud server, the terminal device and the cloud server can perform different reconstructions.

[0100] After the cloud server receives the compressed code stream, the cloud server obtains the metadata from the code stream, and then the cloud server calls the model to obtain a data decoding model, and obtains the image features from the compressed information through the data decoding model. Then the cloud server reconstructs using the cloud-side reconstruction model to obtain a cloud-side reconstructed image. Because the cloud server has stronger computing power, the cloud server can perform more complex image reconstruction operations.

[0101] In the embodiments of the present application, the terminal device and the cloud server can use multiple ways to perform model consistency negotiation in the model preparation stage. For example, the terminal device and the cloud server perform offline model consistency negotiation, or the terminal device and the cloud server perform online model consistency negotiation.

[0102] Specifically, in offline model consistency negotiation, the terminal device and the cloud server respectively store the resource information of the two devices, so that the terminal device and the cloud server can complete offline consistency negotiation through the resource information of each device. The terminal device and the cloud server can achieve consistent end-to-cloud computing.

[0103] For example, the terminal device and the cloud server can perform online model consistency negotiation, and the terminal device and the cloud server can perform online real-time interaction. Detection can be performed from a model with the highest quantization calculation precision, to detect whether the operators of the two devices support the highest quantization precision. Detection is sequentially performed on each quantization calculation precision. When the terminal device and the cloud server each support multiple quantization calculation precisions, the quantization calculation precision with the best model output effect can be selected. After the detection of the quantization calculation precision is completed, the terminal device and the cloud server can also negotiate the implementation manner, for example, both turn on or turn off the acceleration algorithm optimization, and then use a performance analysis tool to measure the execution time of each model to determine the implementation manner suitable for both the terminal device and the cloud server.

[0104] In the embodiment of the application, when the terminal device performs data compression, image data information features can be extracted to obtain one or more feature data, for example, the feature data can be a feature map, for example, a first feature map and a second feature map, wherein the second feature map is obtained by further feature extraction of the first feature map by the model. Then the terminal device can quantize the second feature map to obtain a quantized second feature map. For example, the quantized second feature map can include a convolution layer, an activation layer and a deconvolution layer, which will be described in detail in subsequent FIG. 7 and the corresponding textual part.

[0105] Then the quantized second feature map is entropy encoded to obtain a code stream corresponding to the second feature map. In addition, the data encoding model determined in advance by the terminal device in the model preparation stage can also perform context prediction on the quantized second feature map to obtain encoding statistical information. For example, the encoding statistical information can include a mean and a scale parameter. The first feature map can be entropy encoded according to the encoding statistical information to obtain a code stream corresponding to the first feature map. Finally, the terminal device can package and compress the code stream corresponding to the second feature map, the code stream corresponding to the first feature map and the metadata to obtain a compressed code stream.

[0106] After the terminal device generates the compressed code stream, the terminal device can send the compressed code stream to the cloud server, which can be a sending user terminal. The cloud server receives the compressed code stream and then decodes it using a data decoding model determined in advance by the cloud server to obtain feature data. Then, according to the computing power of the cloud server, a cloud-side reconstruction model of the cloud server is determined, which can be used to reconstruct according to the feature data to obtain a cloud-side reconstruction map. Finally, the cloud server can send the cloud-side reconstruction map to the receiving user terminal.

[0107] Next, the data decompression process is described. First, the compressed code stream is obtained, which includes compressed information and metadata. According to the metadata, the model in the model pool is called to determine the data decoding model. The compressed information is entropy decoded according to the model prior information to obtain the entropy decoded second feature map. The context of the entropy decoded second feature map is predicted according to the data decoding model to obtain the decoding statistical information. The first feature map is determined from the code stream according to the decoding statistical information. Finally, the first feature map is reconstructed to obtain the reconstructed pixel bitmap. For example, an image reconstruction model can be used for reconstruction. For different computing power of terminal devices and cloud servers, different reconstruction methods can be used. For example, for the terminal side with limited computing resources, a lightweight neural network model (reconstruction method 1) can be selected for image reconstruction. For the cloud side with strong computing resources, a larger neural network model (reconstruction method 2) can be used for reconstruction.

[0108] In some embodiments of the present application, the terminal device and the cloud server each use different image reconstruction models. The semantic consistency of the terminal device and the cloud server each using different image reconstruction models can be ensured by offline training of the neural network.

[0109] In the embodiments of the present application, an image enhancement framework facing key regions can be used in the image reconstruction model. The key regions can be divided into face or text, and landscape scenes. Therefore, the image enhancement framework in the embodiments of the present application can include an image enhancement module facing face or text, and an image enhancement module facing landscape scenes. The main difference between the image enhancement module facing face or text and the image enhancement module facing landscape scenes is the selection of the training strategy. The structures of the neural networks of the two modules use the same structure, and the different training strategies affect the different neural network weights.

[0110] As known from the foregoing examples, in the embodiments of the present application, by adaptively selecting the model with the best precision between the terminal device and the cloud server, the inconsistency problem of AI encoding and decoding technology between different platforms is solved. Not only the original picture information is preserved, but also the information loss or abnormal texture caused by the encoding and decoding difference is avoided. In addition, in the embodiments of the present application, by using a more complex reconstruction method on the cloud server side with stronger computing resources, the embodiments of the present application realize the targeted enhancement of key regions while maintaining semantic consistency, so that the key regions are clearer. For example, the key regions can be face, text region, or landscape scene region. The embodiments of the present application can realize the semantic consistent asymmetric size model cooperation.

[0111] Based on the above data processing platform, as shown in FIG. 3, the data processing method executed by the terminal device can include steps 301-304.

[0112] 301, performing feature extraction on the first data to be encoded to obtain first feature data;

[0113] 302, performing context prediction on the first feature data using a first data encoding model indicated by first metadata to obtain encoding statistics, the first data encoding model being a data encoding model determined by the terminal device and the cloud server when performing consistency negotiation before encoding the first data, and a data quantization precision of the first data encoding model indicated by the first metadata matching a data quantization precision of a first data decoding model used by the cloud server;

[0114] 303, encoding the first feature data according to the encoding statistics to obtain a first code stream;

[0115] 304, compressing the first metadata and the first code stream to obtain a compressed code stream, the compressed code stream being used for sending to the cloud server.

[0116] In some embodiments of the present application, the method further comprises:

[0117] The terminal device respectively acquires resource information of the terminal device and resource information of the cloud server;

[0118] According to the resource information of the terminal device and the resource information of the cloud server, consistency detection is performed on whether the terminal device and the cloud server respectively support quantization precision, and a consistency detection result is obtained;

[0119] According to the consistency detection result, the first data encoding model is determined from a model set of the terminal device;

[0120] According to the data quantization precision of the first data encoding model, the first metadata is generated.

[0121] In some embodiments of the present application, the first metadata is generated according to the data quantization precision of the first data encoding model, comprising:

[0122] According to the data quantization precision of the first data encoding model, an acceleration algorithm implementation manner of the first data encoding model is determined;

[0123] According to the data quantization precision of the first data encoding model and the acceleration algorithm implementation manner of the first data encoding model, the first metadata is generated.

[0124] In some embodiments of the present application, the consistency detection on whether the terminal device and the cloud server respectively support quantization precision according to the resource information of the terminal device and the resource information of the cloud server comprises:

[0125] perform offline consistency detection according to the resource information of the terminal device and the resource information of the cloud server; or

[0126] perform online consistency detection by real-time interaction with the cloud server according to the resource information of the terminal device and the resource information of the cloud server.

[0127] Based on the above data processing platform, the data processing method performed by the cloud server can include steps 401-404.

[0128] 401. Decoding the compressed code stream from the terminal device to obtain decoded second feature data;

[0129] 402. Context prediction of the second feature data using a first data decoding model indicated by the second metadata to obtain decoding statistical information, the first data decoding model being a data decoding model determined during consistency negotiation before the cloud server obtains the compressed code stream, and the data quantization precision of the first data decoding model indicated by the second metadata matching the data quantization precision of the first data encoding model used by the terminal device;

[0130] 403. Decoding third feature data from the compressed code stream according to the decoding statistical information;

[0131] 404. Reconstructing the third feature data according to the reconstruction model of the cloud server to obtain reconstructed data.

[0132] To better understand and implement the above scheme of the embodiments of the present application, the following examples of corresponding application scenarios are used for specific description.

[0133] The computing resources of the mobile terminal and the cloud server are different, so using the same encoding and decoding algorithm can cause mismatch. For example, the computing precision of the cloud and the mobile terminal is different, causing the results generated by the AI encoder and the AI decoder to have floating point errors. This can cause the decoding information required when the code stream is decompressed to be inconsistent with the encoding information when compressed, thus causing encoding and decoding mismatch. In the embodiments of the present application, the mobile terminal and the cloud server can perform model consistency negotiation before data encoding, thereby solving the problem of encoding and decoding mismatch.

[0134] The embodiments of the present application solve the problem of codec mismatch, and can further improve the picture reconstruction quality by using devices with greater computing power under the premise of correctly decoding the image. Cloud servers usually have more powerful computing power than mobile terminals and can perform more complex image reconstruction operations. The embodiments of the present application can maximize the difference between cloud servers and mobile terminals, ensure that the image can be correctly decoded, and improve the efficiency of image recovery and enhancement.

[0135] The embodiments of the present application provide an asymmetric AI codec scheme for end-to-cloud cooperation, which can realize cross-device codec consistency and asymmetric model size of end-to-cloud to assist image enhancement.

[0136] The embodiments of the present application mainly include:

[0137] Cross-platform function calculation consistency technology: by offline deploying a multi-precision calculation model, offline generating precision consistency prior information, and online handshake based on prior information comparison or based on basic mathematical operator calculation detection, selecting the optimal precision model suitable for calculation consistency, ensuring calculation consistency while reducing calculation precision loss.

[0138] Asymmetric size model semantic consistency technology: after ensuring the decoding module of calculation consistency, different calculation platforms adapt to different complexity of generated enhancement operators, and improve the decoding quality on the basis of similar semantics, wherein the different calculation platforms specifically refer to different devices, such as mobile terminals and cloud servers.

[0139] The embodiments of the present application are mainly applied to the terminal communication scene of end-to-cloud cooperation, as shown in FIG. 4, and the execution process of the communication scene mainly includes the following steps:

[0140] The terminal device can perform subsequent steps S01 to S05, and the cloud server can perform subsequent steps S11 to S14.

[0141] S01, model preparation:

[0142] The terminal device selects a model to be called. In the embodiments of the present application, the terminal device can select the model by the following way, the terminal and the cloud perform offline or online data precision agreement and precision implementation check, then select the image codec model in the pre-deployed model pool, and the precision agreement and implementation way are written in the metadata, the selected model from the model pool can be used in the subsequent image codec.

[0143] In the embodiments of the present application, before model preparation, the terminal device and the cloud server need to perform pre-deployment of the model pool. In the model pool of the terminal and the cloud, three models of different precisions are deployed: fp32, fp16 and int8. These models are determined according to the requirements of the application scenario and the resource constraints. Among them, the precision of the fp16 and int8 models is obtained by converting the original fp32 model using a quantization tool. For example, using int8 precision means that the weights originally in the form of fp32 in the model are converted to integers in the form of int8 through the quantization process.

[0144] Regardless of the terminal or the cloud, each precision model provides two implementation methods: one is to enable acceleration algorithm optimization, such as enabling the winograd acceleration algorithm, and the other is to disable acceleration algorithm optimization. The choice of these two implementation methods is based on extensive experimental data and performance evaluation.

[0145] The difference between the model pool of the terminal and the cloud is that the model pool of the terminal uses a model format suitable for mobile AI frameworks, such as Tensor Flow Lite, while the model pool of the cloud uses a model format for server-side AI frameworks, such as Tensor Flow. Such arrangements ensure that the terminal device and the cloud server can achieve the best efficiency and accuracy in their respective running environments.

[0146] S02, reading of data:

[0147] Among them, the terminal device can read image data, for example, a user selects a picture from the gallery, and the terminal device converts the format of the picture into a bitmap image. The bitmap image can also be called a dot matrix image or a raster image, and the bitmap image is obtained by multiple picture elements.

[0148] S03, model calling and encoding:

[0149] The terminal device inputs the read bitmap image into an image encoder, for example, the image encoder can be an AI encoder. The bitmap image is compressed to a specific code rate by the image encoder to obtain a compressed code stream, and the compressed code rate is not limited in the embodiments of the present application.

[0150] In the embodiments of the present application, the image encoder is the model of the corresponding precision selected in the model preparation stage of step S01.

[0151] S04, sending compressed information:

[0152] After obtaining the compressed code stream, the terminal device can send the compressed code stream to the cloud server.

[0153] In the embodiments of the present application, the header file of the transmitted code stream contains the metadata of the model preparation stage. The compressed code stream is transmitted to the cloud server of the service provider through a network. For example, the transmission of the code stream can be performed in a wired or wireless manner, such as through a telephone line, a satellite connection, or a cellular network, which is not limited herein.

[0154] In an application scenario of the embodiments of the present application, since the user needs to preview the compressed picture, the terminal device side and the cloud server side both perform the decompression step. The terminal device can perform decoding and reconstruction to obtain a reconstructed picture, which can be used for user preview. The reconstructed picture generated by the cloud server is used for receiving user reception.

[0155] It should be noted that the decoding algorithms performed by the terminal side and the cloud side in the embodiments of the present application are completely the same. However, the reconstruction methods used are different.

[0156] Decompression includes two steps of decoding and reconstruction. The decoding algorithms used by the terminal side and the cloud side are completely the same, which is to ensure that consistent feature maps can be correctly decoded in the terminal or the cloud. However, the process of reconstructing the feature maps into pixel maps can be different in the terminal and the cloud due to different device resources.

[0157] The decompression process of the terminal device is described below.

[0158] S05, decoding:

[0159] For example, the terminal device uses an image decoder to decode the compressed code stream to obtain image features. The image decoder is a model with corresponding precision selected in the model preparation stage.

[0160] S06, reconstruction:

[0161] In the embodiments of the present application, the image decoder is a model with corresponding precision selected in the model preparation stage. After obtaining the image features through decoding, the terminal device can also generate an end-side reconstructed picture through a reconstruction model. For example, the end-side reconstructed picture can include a pixel bitmap.

[0162] After the terminal device generates the end-side reconstructed picture through the reconstruction model, the end-side reconstructed picture can be displayed to enable the user to preview the end-side reconstructed picture.

[0163] For example, the features obtained through decoding in the above steps generate an end-side reconstructed picture through an end-side reconstruction model end-side reconstructed picture for user preview.

[0164] Next, the decompression process of the cloud server is described, which mainly includes the following steps:

[0165] S11, receiving compressed information:

[0166] The cloud server can receive the compressed code stream from the terminal device through interaction with the terminal device. The compressed code stream can include compression information.

[0167] S12, model calling:

[0168] The cloud server determines the metadata representing the agreed precision and implementation manner according to the header file in the code stream. The cloud server selects a model in the model pool deployed in advance according to the metadata. The model is subsequently used in the subsequent image decoder.

[0169] It should be noted that in the embodiments of the present application, the model pool used by the terminal device and the cloud server is different. The difference lies in that the AI frameworks used by the terminal and the cloud are different, so the model formats are different, but the model precision and implementation method in the model pool of the terminal and the cloud can be the same.

[0170] S13, decoding:

[0171] On the cloud server, the received image data is decoded into image features by the image decoder.

[0172] S14, reconstruction:

[0173] The cloud server can also generate a cloud-side reconstruction map through a reconstruction model. For example, the cloud-side reconstruction map can include a pixel bitmap.

[0174] After the cloud server generates the cloud-side reconstruction map through the reconstruction model, the cloud server can send the cloud-side reconstruction map to the terminal device of the receiving user, so that the receiving user can receive the cloud-side reconstruction map.

[0175] For example, the decoded features are reconstructed into a cloud-side reconstruction map by a reconstruction model on the cloud side The cloud-side reconstruction map is sent to the receiving user through the communication device of the cloud server.

[0176] Next, the core network element, device module, and product implementation of the embodiments of the present application are described. The embodiments of the present application can realize software-level optimization for terminal devices and cloud servers. The model preparation, data compression, and data decompression stages are described respectively.

[0177] The embodiments of the present application can realize cross-platform function calculation consistency, involving the model preparation, data compression, and data decompression stages. In addition, the embodiments of the present application can also realize asymmetric model collaboration, involving the data decompression stage.

[0178] The terminal device can include a context prediction module. In the AI codec model, the context prediction module can affect the end-to-cloud consistency result. In the model preparation stage, the data precision of the context prediction module and the implementation mode of the operator need to be determined according to the end-to-cloud information.

[0179] As shown in FIG. 5, in the model preparation stage, the specific process is as follows:

[0180] S21, the terminal device reads the respective resource information of the terminal device and the cloud server.

[0181] Among them, the resource information mainly includes at least one of the following four kinds of information:

[0182] 1) hardware information of the terminal device and the cloud server, for example, the hardware information can include neural processing unit (NPU) parameters, central processing unit (CPU) parameters, graphics processing unit (GPU) parameters, and memory size.

[0183] 2) AI framework and version information used by the terminal device and the cloud server. For example, the AI framework of the cloud server includes Pytorch, tensorflow, and mindspore. The AI framework of the terminal device includes Pytorch mobile, tensorflow lite, and mindspore lite. The corresponding version information refers to the version number of the currently installed AI framework of the device. Taking pytorch as an example, the version number is in the following format: v2.2.0.

[0184] 3) calculation precision supported by the AI framework for the operators involved in the model.

[0185] Among them, the present application embodiment involves AI codec, and the operator involved is a plurality of types of operators of neural network, for example, the operator can include convolution, deconvolution, activation operator ReLU (linear rectifier function), etc., which are not limited here.

[0186] For example, the process of offline pre-checking the end-to-cloud calculation consistency includes:

[0187] a) determine the test function, for example, the test function f = Deconv * Conv, where Deconv is a deconvolution with a kernel size of 3*3, an input channel of 3, and an output channel of 3, and Conv is a convolution with a kernel size of 3*3, an input channel of 3, and an output channel of 3.

[0188] b). Determine the precision of the test and the operator implementation. If the int8 precision is selected, the winograd acceleration method is used.

[0189] c). Preprocess the test function, use the quantization tool to quantize the convolution and deconvolution weights in the test function to int8, and enable the winograd acceleration implementation.

[0190] d). Select a test picture, execute the processed test function on the two devices respectively, and check whether the output results of the two devices are consistent.

[0191] 4), the implementation method of AI framework to support the operator of model.

[0192] S22, the terminal device and the cloud server respectively judge whether the end cloud device information is recorded in the offline pre-check result. The end cloud device information refers to the resource information of the terminal device and the cloud server respectively.

[0193] Among them, the offline pre-check refers to the consistency comparison of the chip or computing platform in advance.

[0194] S23, if the end cloud device information is recorded in the offline pre-check result, it is judged according to the pre-check result whether the two devices can realize the consistency of end cloud computing. If the end cloud computing consistency can be realized, the precision and implementation set used in the offline pre-check are written into the metadata. The corresponding pre-deployed model in the model pool is called. If the end cloud computing consistency cannot be realized, the process ends and returns an error message.

[0195] S24, if the end cloud device information is not in the offline pre-check result, it is necessary to detect whether the device can support consistency and the precision and implementation method supported by the device online, and execute the subsequent step 5. If the end cloud device does not support online detection, the process ends and returns an error message.

[0196] S25, if the end cloud device supports online detection, the highest precision that can be supported is selected to start the test. It is checked whether there is the same operator implementation under the highest precision of the end cloud two devices. If there is no same operator, the previous step is returned, and the next low precision is selected to start the test. If there is the same operator implementation, the next step is entered. The same test experiment is executed on the end cloud two devices, and it is compared whether the test results are the same. For example, in the case that the end cloud device supports online mutual transmission of information, it is necessary to mutually send the experimental results.

[0197] If the test results are the same and all the sets of implementations meet the standard, the performance indicators of the evaluation test are tested, and the optimal one is selected. The above precision requirements and corresponding operator implementations are written into the metadata. For example, the specific scheme for selecting the optimal model is as follows: first, the same test data set is input into multiple models with different implementation modes. These models differ in operator implementation, such as turning on or off the Winograd algorithm optimization. Then, the execution time of each model is measured using a performance analysis tool. Finally, the execution times are compared, and the model with the shortest execution time is selected as the optimal model. This method ensures that the model with the highest efficiency can be selected while ensuring the accuracy of the model.

[0198] The terminal device and the cloud server perform end-cloud offline deployment according to the metadata, respectively, to generate models corresponding to the quantization precision and implementation. If the test results are different, return to the precision selection step and select the next low precision for testing.

[0199] The above-mentioned multiple sets refer to multiple operator implementation modes under the same precision. For example, there are multiple operator implementations for convolution operations, such as Fast Fourier Transform, Winograd transform, or direct calculation.

[0200] S26, if the current test precision is the lowest precision that can be achieved, and there is no same operator implementation or the test results are different, the process ends and an error message is returned.

[0201] As shown in FIG. 6, the specific process of the data compression phase in the embodiment of the application is explained as follows, including:

[0202] S31, data reading:

[0203] The terminal device converts the input image into a bitmap.

[0204] S32, feature extraction:

[0205] The bitmap is transmitted to the feature extraction module of the terminal device, and the first feature map y and the second feature map z can be obtained through the feature extraction module.

[0206] It should be noted that the number of feature maps is determined by the design of the model. In the embodiment of the application, the number of models is not limited, for example, the current implemented model has only two feature maps, but other similar models may have a third feature map, etc.

[0207] The main difference between the first feature map y and the second feature map z is that the second feature map is further extracted from the first feature map by the model. For example, there are two functions f_1 and f_2 in the feature extraction module, the input picture is x, the first feature map y = f_1(x), and the second feature map z = f_2(y) = f_2(f_1(x)).

[0208] S33, feature quantization:

[0209] The terminal device quantizes the second feature map into an integer to obtain z_hat.

[0210] In the embodiments of the present application, before image encoding, the feature map will be quantized. According to the generation sequence of the first feature map and the second feature map, the second feature map will be quantized and encoded first. Then in the step of context modeling, the first feature map will be quantized.

[0211] S34, entropy encoding:

[0212] The terminal device performs entropy encoding on the quantized second feature map z_hat to obtain the code stream z_string.

[0213] S35, context modeling:

[0214] The quantized second feature map z_hat is input into the context prediction module with the predetermined precision to obtain the encoding statistical information, which can include the mean m and the scale parameter scale.

[0215] For example, the mean m and the scale parameter scale are obtained by z_hat through two neural networks, a mean calculation module and a scale calculation module, respectively.

[0216] S36, generating code stream:

[0217] The terminal device obtains the quantized first feature map residual y_r = Quantize (y-m) using the mean m.

[0218] Then, the y_r is entropy encoded using the scale parameter scale to obtain the first code stream y_string.

[0219] S37, data packaging and sending:

[0220] The metadata obtained in the model preparation stage is written into the header file, and the two code streams z_string and y_string are data packaged to generate the compressed code stream, which is sent to the cloud server.

[0221] As shown in FIG. 6, the specific process of the data decompression stage in the embodiments of the present application is explained as follows, including:

[0222] S41, compressed file reading:

[0223] The cloud server reads the compressed bitstream and obtains a compressed file. This file consists of two parts: one part is the compressed bitstream data, z_string and y_string, and the other part is a header file, which contains metadata representing precision information and implementation details.

[0224] S42, Model Selection Invocation:

[0225] The cloud server selects the corresponding model from the model pool based on the metadata content and then invokes it.

[0226] S43, Entropy Decoding:

[0227] The cloud server performs entropy decoding on the second bitstream z_string based on the model's prior information to obtain the decoded second feature map z_hat.

[0228] S44, Context Prediction:

[0229] The cloud server inputs the decoded second feature map z_hat into a context prediction module with a predetermined precision to predict the statistical information mean m and scale parameter. Using the scale parameter, the residual y_r is decoded from the bitstream y_string. The first feature map y is obtained by applying Dequantize(y_r) + m = y_hat. y_hat represents an approximation of the first feature map, obtained by adding the mean m to the decompressed residual y_r.

[0230] It should be noted that, regardless of whether it is on the edge or the cloud, the cross-platform function computation consistency technology can ensure that the scale parameter obtained after the context prediction module is the same. Therefore, the residual y_r obtained after decoding is strictly the same as the y_r before encoding.

[0231] S45, Image Reconstruction:

[0232] The cloud server performs image reconstruction on the first feature map y to obtain the reconstructed pixel bitmap.

[0233] In this embodiment, different reconstruction methods can be used depending on the computing power of the terminal device and the cloud server. For example, a lightweight neural network model (referred to as reconstruction method 1) can be selected for image reconstruction on a terminal device with limited computing resources. On a cloud server with powerful computing resources, a larger neural network model (referred to as reconstruction method 2) can be used for reconstruction, thus enabling asymmetric model-size collaboration.

[0234] It should be noted that the decompression in the embodiments of the present application includes two steps: decoding and reconstruction. In the decoding step, to ensure the consistency of decoding, the information (i.e., the scale parameter scale) obtained by the edge side and the cloud side for decoding must be exactly the same. Therefore, in the decoding step, the model structure and weights used by the edge side and the cloud side must be the same. In the reconstruction step, for the neural network module used in this step, the edge cloud can select according to resources, such as the edge side using a lightweight model, and the cloud side using a large model.

[0235] The semantic consistency of the reconstruction method 1 and the reconstruction method 2 described above is guaranteed by the offline training of the neural network. The specific method is as follows:

[0236] a) Obtain a training data set: the data set contains N pictures.

[0237] b) Determine the weights of the network to be updated: the weights in the fixed feature extraction module and the context modeling module are not updated, and only the weights in the reconstruction module 1 and the reconstruction module 2 are updated.

[0238] c) Network inference to obtain a reconstructed picture: input the picture x into the compression and decompression module described above to obtain the decoded neural network feature map. The neural network feature map is passed through the reconstruction module 1 and the reconstruction module 2 respectively to obtain the edge side reconstruction picture and the cloud side reconstruction picture

[0239] e) Calculate the loss function:

[0240] The two reconstructed pictures obtained are input into the semantic consistency loss function together with the original picture in the data set for calculation. The loss function can be specifically:

[0241] wherein, represents the loss function between x and represents the loss function between x and represents the loss function between x and represents the loss function between x and represents the loss function between and represents the loss function between and, λ1 and λ2 represent weights. min represents the minimum value of and . represents the edge side reconstruction picture when the backpropagation gradient is stopped, represents the cloud side reconstruction picture when the backpropagation gradient is stopped.

[0242] wherein, ​

[0243] Finally, the model updates the weights in reconstruction module 1 and reconstruction module 2 according to the loss function by backpropagation of gradients.

[0244] Next, the application scenarios of the embodiments of the present application are described. The shooting of high-resolution images and the communication transmission cost are high. In the case of limited transmission bandwidth, image compression is performed on the terminal device, and the compressed signal is sent to the cloud server. The data is decoded and enhanced in the cloud server with more computing power, and returned to the user terminal for transmission. The problem of limited terminal computing power and limited transmission bandwidth can be solved.

[0245] The embodiments of the present application can realize signal recovery and enhancement of the cloud side to help the end side. Two problems need to be considered:

[0246] 1. The coding and decoding consistency across platforms means that the same data is encoded on the end side, decoded by the cloud side, and still maintains the same content as the original data. Inconsistent coding and decoding will cause the decoded information to deviate from the information before encoding, resulting in information errors. Only by solving this problem and decoding the correct information can the correct semantic image be reconstructed to help the subsequent image enhancement technology to better reconstruct the key area. If the coding and decoding are inconsistent, there will be garbled codes, so that the subsequent image enhancement technology cannot play an enhancement role.

[0247] 2. Asymmetric size model coordination technology: decoding requires the end and the cloud to select the same network structure and precision, but the reconstruction part can select different reconstruction modules according to the device computing power. On the premise of ensuring the consistency of coding and decoding, the large computing power of the cloud side can be maximally utilized during image reconstruction, and a complex model can be used to generate and enhance the image, thereby improving the reconstruction effect of the cloud side.

[0248] The technical solutions of the embodiments of the present application include three key stages of model preparation, image compression, and image decompression.

[0249] 1. Model preparation, which can realize the function calculation consistency technology across platforms:

[0250] In this embodiment, the device information of the mobile phone and the device information of the cloud server are recorded in the offline pre-check result. For example, the device information of the mobile phone side can include that the system-level chip of the mobile phone side is Kirin 9000, and the full-scene AI computing framework is Mindspore lite framework. The device information of the cloud server can include that the chip is Ascend chip 910, and the AI computing framework is pytorch framework. The mobile phone and the cloud server can agree to use an int8 precision quantization model. In this embodiment of the application, the scale parameter calculation module in the context prediction module is involved, and it is agreed that the convolution implementation method of the end-to-cloud model is to close the winograd fast Fourier acceleration method. It is found through experimental results that the experimental results of using the winograd acceleration method in the mindspore lite framework are inconsistent with the experimental results of using the winograd acceleration method in the pytorch, and therefore need to be closed. In this embodiment, the network structure of the scale parameter calculation module to be quantized is shown in FIG. 7.

[0251] FIG. 7 is an example of a possible network structure of the scale parameter calculation module. The quantized second feature map is the input information of the module. In the second feature map, it can include:

[0252] Convolution layer: The main function of the convolution layer is to extract the features of the input data. Through the convolution operation of the convolution kernel obtained by training on the input data, local features such as edges and corner points can be captured. After the convolution operation, a nonlinear activation function operation such as ReLU is performed.

[0253] Activation layer: The activation layer performs a nonlinear transformation on the output of the convolution layer through an activation function, so that the network model has the ability to process nonlinear problems. For example, the activation functions that can be used in this embodiment include ReLU, Sigmoid, and Tanh.

[0254] Deconvolution layer: The deconvolution layer is used to increase the size of the feature map. The deconvolution layer performs an operation opposite to convolution, mapping the feature map back to a higher resolution space.

[0255] These layers work together to enable the convolutional neural network to learn useful feature representations from raw data.

[0256] 2. Compression of images:

[0257] As shown in FIG. 8, the terminal device performs a data compression process. The terminal device performs step 1: feature extraction to obtain a first feature map y and a second feature map z. The first and second feature maps are abbreviated as the first and second feature maps. The second feature map z is quantized to obtain a quantized second feature map Step 2: Entropy coding is performed on the second feature map to obtain a second code stream z_string.

[0258] As shown in FIG. 9, the internal structure of the mean value calculation module is compression. The input y and z are used to obtain the predicted mean value m through a neural network, and then the output result y_r is obtained through Quantize(y-m)=y_r.

[0259] In addition, the terminal device can also perform image reconstruction according to the transmission code stream to output a reconstructed picture.

[0260] 3. Decompression of the image:

[0261] As shown in FIG. 10, the cloud server receives the code stream sent by the terminal device. The cloud server performs step 1: entropy decoding is performed on the second code stream z_string to obtain the second feature map Step 2: Context modeling is performed to calculate the required statistical information, and entropy decoding is performed on the first code stream y_string to obtain the first feature map. Specifically, the second feature map The scale parameter calculation module is inputted, and the scale parameter scales is obtained through the scale parameter calculation module. Entropy decoding is performed on the received first code stream y_string according to the scale parameter to obtain y_r. y_r and the second feature map The mean value calculation module is inputted, and the cloud side reconstructed picture is outputted

[0262] As shown in FIG. 11, the internal structure of the mean value calculation module is decompression. The input y and z are used to obtain the predicted mean value m through a neural network, and then the approximate first feature map

[0263] 4. Image reconstruction in decompression:

[0264] By using the stronger computing power and memory resources of the cloud server, and aiming at the face and text regions sensitive to user vision, the image enhancement framework for key regions is used in the image reconstructor on the cloud side, and the face and text regions are enhanced according to the image category.

[0265] As shown in FIG. 12a, the specific process is as follows: the first feature map obtained after decoding is input into the feature classifier to obtain a classification result, which can indicate whether there is a face and text region. If the classification result is that neither a face nor a text region exists, the image enhancement module for a landscape scene is called. If there is a face or a text region, the image enhancement module for a face or text is called.

[0266] In the present embodiment, the main difference between the image enhancement module for a face or text and the image enhancement module for a landscape scene lies in the selection of the training strategy. The structures of the neural networks of the two modules use the same structure, and the different training strategies affect the different neural network weights. In the present embodiment, the neural network in the image enhancement module is selected as a U-shaped neural network (UNet).

[0267] A landscape image contains extensive backgrounds and natural elements such as mountains, lakes, and trees. In the training, the present embodiment uses a generative adversarial loss function (GAN) to enhance the landscape image. GAN helps to improve the overall quality of the image. GAN can capture the details and textures in the natural environment, making it more realistic.

[0268] A face and a text region have particularity for image enhancement. The image enhancement module for a face or text in the present embodiment can maintain the clarity of these regions while avoiding distortion. GAN can cause distortion in these regions, so in these regions, the present embodiment uses an L1 loss function to avoid using GAN, which helps to make the enhanced image clearer while maintaining the structural authenticity.

[0269] Therefore, for the two different scenes of a face, a text, and a landscape, the present embodiment uses different training strategies to ensure that the optimal enhancement effect can be achieved in different scenes.

[0270] As shown in FIG. 12b, it is a flowchart of the enhancement module for a face / text. The training strategy of the image enhancement module for a face or text is as follows: for a target region such as a face or a text, an L1 loss function is selected to train the region during training, so that the reconstruction result is clearer and the structural authenticity is maintained. For other regions except the target region, an L2 loss function and a generative adversarial loss function are used. The specific steps are as follows:

[0271] a) Obtain a face and text training data set: the data set contains N pictures and labeled face or text mask boxes.

[0272] b) Obtain a reconstructed picture by network inference: input the picture into the image enhancement network to obtain an enhanced picture

[0273] c) Calculate the loss function for face or text enhancement: the reconstructed picture and the original picture x in the dataset, the labeled face or text mask frame mask, are input into the loss function for calculation. The loss function is defined as:

[0274] Loss function

[0275] Where, loss pixel is the L2 loss function,

[0276] loss obj is the L1 loss function, loss gan is the generative adversarial loss function.

[0277] d) The model updates the weights according to the loss function and backpropagation gradient.

[0278] As shown in FIG. 12c, it is a flow chart of the landscape enhancement module. The training strategy of the landscape-oriented image enhancement module is: for pictures that are all landscapes, select L2 loss function and generative adversarial loss function for the whole picture during training to maximize the enhancement effect on landscapes. The specific steps are as follows:

[0279] a) Obtain the landscape training dataset: the landscape dataset contains N pictures.

[0280] b) Network inference to obtain the reconstructed picture: input the picture into the image enhancement network to obtain the enhanced picture

[0281] c) Calculate the loss function for landscape: the reconstructed picture and the original picture x in the dataset are input into the loss function for calculation. The loss function used is:

[0282] Loss function

[0283] Where, loss pixel is the L2 loss function, loss gan is the generative adversarial loss function.

[0284] d) The model updates the weights according to the loss function and backpropagation gradient.

[0285] As shown in FIG. 13, it is an example diagram of a network structure of a possible feature classifier. For example, the first feature map y includes a convolution layer, an activation layer and a deconvolution layer, each layer has different dimensions of data, for example, the data dimension of the first convolution layer is (256, 16, 12), etc. Through different fully connected layers, the face classification result and the text classification result can be output respectively, and the dimension of the data after passing through each layer will change.

[0286] Based on the above examples, it can be known that the embodiment of the application can realize cross-platform function calculation consistency technology and asymmetric size model cooperation. In the embodiment, the same quantization precision and precision can be agreed through end-to-cloud according to different device resources, the model quantization precision is adaptively selected, the coding and decoding consistency is ensured, in addition, the greater computing power of the cloud server can be used, and a more complex reconstruction method is used to specifically enhance the key areas such as face and text. The reconstruction method can include the aforementioned image enhancement framework for key areas.

[0287] The embodiment of the application supports adaptive adjustment of quantization precision and implementation strategy according to different end-to-cloud device information. The optimal precision model suitable for the scene calculation consistency is selected, the calculation consistency is ensured, and the calculation precision loss is reduced. The embodiment of the application supports adaptation of image reconstruction and enhancement operators of different complexities according to different end-to-cloud device information, improves the decoding quality on the basis of ensuring similar semantics.

[0288] Compared with the AI encoder and decoder of the prior art, in solving the cross-platform coding and decoding consistency, the embodiment of the application adaptively adjusts the quantization precision and implementation method of the model according to the different resources of multiple devices, and maximizes the loss caused by model quantization. Compared with the AI encoder and decoder of the prior art, the embodiment of the application uses the same image reconstruction method on multiple devices, and uses the asymmetric size model cooperation technology according to the different device resources, uses the image reconstruction and enhancement method with more complex structure on the cloud server with strong computing resources, and maximizes the use of the resources of the cloud server to optimize the image reconstruction quality.

[0289] Cross-platform function calculation consistency technology: through offline deployment of multi-precision calculation models, offline generation of precision consistency prior information, and online handshake based on prior information comparison or based on basic mathematical operator calculation detection, the optimal precision model suitable for the scene calculation consistency is selected, the calculation consistency is ensured, and the calculation precision loss is reduced.

[0290] Asymmetric size model cooperation deployment architecture: after the decoding module that ensures the calculation consistency, different complexity generation enhancement operators are adapted on different computing platforms, and the decoding quality is improved on the basis of ensuring similar semantics.

[0291] The key technical points of the embodiment of the application correspond to the beneficial effects:

[0292] Cross-platform function calculation consistency: the embodiments of the present application solve the inconsistency problem of AI encoding and decoding technology between different platforms by adaptively selecting the best precision model between the terminal device and the cloud service. Not only the original picture information is preserved, but also the information loss or abnormal texture caused by encoding and decoding difference is avoided.

[0293] As shown in FIG. 14a, the reconstructed image obtained by decompression and reconstruction on the cloud server GPU without using the encoding and decoding consistency technology after the CPU compression on the mobile phone side, the image has information loss. As shown in FIG. 14b, the reconstructed image obtained by decompression and reconstruction on the cloud server GPU using the encoding and decoding consistency technology, the cloud server side can reconstruct a clear image.

[0294] As shown in FIG. 14c, the reconstructed image obtained by decompression and reconstruction on the cloud server GPU without using the encoding and decoding consistency technology after the CPU compression on the mobile phone side, the image has abnormal texture. As shown in FIG. 14d, the reconstructed image obtained by decompression and reconstruction on the cloud server GPU using the encoding and decoding consistency technology, the cloud server side can reconstruct a clear image.

[0295] Semantically consistent asymmetric size model collaboration: by using a more complex reconstruction method on the cloud server side with stronger computing resources, the embodiments of the present application realize the targeted enhancement of the face and text area while maintaining semantic consistency, so that the face and text area are clearer. For example, different computing platforms adapt different complexity of generated enhancement operators. Since the computing power of the mobile phone side is weak, the face or text area of the reconstructed image has low clarity, and the cloud server has stronger computing power, so the decoding quality can be improved on the cloud server side while ensuring semantic consistency.

[0296] It should be noted that the cross-platform function calculation consistency technology provided by the embodiments of the present application can be transplanted into other image encoders, and has easy portability. In addition, the application scenarios of the embodiments of the present application are not limited to image encoding, but also apply to speech and video encoding scenarios, and have matching scene diversity.

[0297] It can be understood that, in the foregoing embodiments, the data processing method is applied to image data processing as an example. For the foregoing generation of bitmaps, feature maps, and entropy encoding, these processes are still applicable to speech and video encoding. The difference is that the data dimensions are different. For example, for speech encoding, the bitmap can be replaced by a one-dimensional audio signal, and the feature map is replaced by a one-dimensional feature map from a two-dimensional feature map. Similarly, in video encoding, the bitmap can be replaced by a three-dimensional video signal, and the feature map can be replaced by a three-dimensional feature map from a two-dimensional feature map.

[0298] The data processing method can be applied to various application scenarios, and each application scenario will be introduced below.

[0299] 1. Internet of Things (IoT): The data processing method of any embodiment of the present application can be applied to the field of Internet of Things, such as smart home, industrial automation, etc. The data processing platform of the terminal device is used to compress IoT data, and the data processing platform of the cloud server is used to decompress IoT data, thereby improving the efficiency of device management and optimization.

[0300] 2. Intelligent manufacturing: The data processing method of any embodiment of the present application can be applied to the field of intelligent manufacturing, such as quality control of production line, equipment maintenance, etc. The data processing platform of the terminal device is used to compress data of intelligent manufacturing, and the data processing platform of the cloud server is used to decompress data of intelligent manufacturing, thereby improving the efficiency of device management and optimization.

[0301] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action order described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0302] In order to better implement the above-mentioned scheme of the embodiments of the present application, the related device for implementing the above-mentioned scheme is also provided.

[0303] Please refer to FIG. 15, the terminal device 1500 provided by the embodiments of the present application can include a receiving module 1502, a processing module 1503 and a sending module 1501, wherein,

[0304] The processing module is configured to perform feature extraction on the first data to be encoded to obtain first feature data.

[0305] The processing module is configured to perform context prediction on the first feature data using a first data encoding model indicated by the first metadata to obtain encoding statistical information, wherein the first data encoding model is a data encoding model determined by the terminal device and the cloud server when performing consistency negotiation before encoding the first data, and the data quantization precision of the first data encoding model indicated by the first metadata matches the data quantization precision of a first data decoding model used by the cloud server.

[0306] The processing module is configured to encode the first feature data according to the encoding statistical information to obtain a first code stream.

[0307] The processing module is configured to compress the first metadata and the first code stream to obtain a compressed code stream, and the compressed code stream is configured to be sent to the cloud server.

[0308] In some embodiments of the present application, the receiving module 1502 is configured to receive the first data to be encoded from an input interface of the terminal.

[0309] In some embodiments of the present application, the sending module 1501 is configured to send the compressed code stream to the cloud server.

[0310] Please refer to FIG. 16, the cloud server 1600 provided by an embodiment of the present application can include a receiving module 1601, a processing module 1602 and a sending module 1603, wherein,

[0311] The processing module 1602 is configured to decode the compressed code stream from the terminal device to obtain decoded second feature data.

[0312] The processing module 1602 is configured to use a first data decoding model indicated by the second metadata to perform context prediction on the second feature data to obtain decoding statistical information, the first data decoding model is a data decoding model determined when consistency negotiation is performed before the cloud server acquires the compressed code stream, and the data quantization precision of the first data decoding model indicated by the second metadata matches the data quantization precision of a first data encoding model used by the terminal device.

[0313] The processing module 1602 is configured to decode third feature data from the compressed code stream according to the decoding statistical information.

[0314] The processing module 1602 is configured to reconstruct the third feature data according to a reconstruction model of the cloud server to obtain reconstructed data.

[0315] In some embodiments of the present application, the receiving module 1601 is configured to receive the compressed code stream from the terminal device.

[0316] In some embodiments of the present application, the sending module 1603 is configured to send the reconstructed data to a receiving user terminal.

[0317] In the embodiments of the present application, the module is an example of a software functional unit, and the data processing apparatus can include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the computing device can be one or more. For example, the data processing apparatus can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region, or can be distributed in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same available zone (AZ), or can be distributed in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0318] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC), or can be distributed in multiple VPCs. Generally, one VPC is set up in one region. Communication between two VPCs in the same region and between VPCs in different regions requires setting up a communication gateway in each VPC to realize interconnection between VPCs through the communication gateway.

[0319] As an example of a hardware functional unit, the module can include at least one computing device, such as a server or the like. Alternatively, the data processing apparatus can also be a device implemented by an application specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented by a complex PLD (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0320] The plurality of computing devices included in the data processing apparatus can be distributed in the same region or in different regions. The plurality of computing devices included in the data processing apparatus can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the data processing apparatus can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0321] Embodiments of the present application also provide a computing device 130. As shown in FIG. 17, the computing device 130 includes a bus 132, a processor 134, a memory 136, and a communication interface 138. The processor 134, the memory 136, and the communication interface 138 communicate with each other through the bus 132. The computing device 130 can be a server or a terminal device. It should be understood that the number of processors and memories in the computing device 130 is not limited by the present application.

[0322] The bus 132 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 17, but it does not mean that there is only one bus or only one type of bus. The bus 134 can include a path for transmitting information between various components (e.g., the memory 136, the processor 134, the communication interface 138) of the computing device 130.

[0323] The processor 134 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0324] The memory 136 can include volatile memory, such as random access memory (RAM) and non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD).

[0325] The executable program code is stored in the memory 136, and the processor 134 executes the executable program code to implement the functions of the aforementioned obtaining module and training module, respectively, so as to implement the data processing method applied to the computing device cluster in the above embodiments. That is, the instructions for executing the data processing method applied to the computing device cluster in the above embodiments are stored on the memory 136.

[0326] The communication interface 138 uses a transceiving module such as, but not limited to, a network interface card and a transceiver to implement the communication between the computing device 130 and other devices or communication networks.

[0327] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0328] As shown in FIG. 18, the computing device cluster includes at least one computing device 130. The memory 136 in one or more computing devices 130 in the computing device cluster can store the same instructions for executing the data processing method.

[0329] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster can also respectively store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 130 can collectively execute the instructions for executing the data processing method.

[0330] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster can store different instructions for respectively executing partial functions of the data processing method. That is, the instructions stored in the memory 136 in different computing devices 130 can implement one or more functions of the obtaining module.

[0331] In some possible implementation, one or more of the computing devices in the cluster of computing devices can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 18 illustrates one possible implementation. As shown in FIG. 18, two computing devices 130A and 130B are connected through a network. Specifically, the computing devices are connected to the network through a communication interface in each of the computing devices. In this type of possible implementation, the memory 136 in the computing device 130A can store instructions for performing the functions of the first processing module. Meanwhile, the memory 136 in the computing device 130B can store instructions for performing the functions of the second processing module. Alternatively, the memory 136 in the computing device 130A can store instructions for performing part of the functions of the second processing module. Meanwhile, the memory 136 in the computing device 130B can store instructions for performing another part of the functions of the second processing module, and so on.

[0332] It should be understood that the functions of the computing device 130A shown in FIG. 19 can also be performed by multiple computing devices 130. Similarly, the functions of the computing device 130B can also be performed by multiple computing devices 130.

[0333] The embodiments of the present application also provide another cluster of computing devices. The connection relationship between the computing devices in the cluster of computing devices can be similar to the connection manners of the clusters of computing devices shown in FIG. 17 and FIG. 18. The difference is that the memory 136 in one or more of the computing devices 130 in the cluster of computing devices can store the same instructions for performing the data processing method.

[0334] In some possible implementation, the memory 136 in one or more of the computing devices 130 in the cluster of computing devices can also respectively store part of the instructions for performing the data processing method. In other words, the combination of one or more of the computing devices 130 can collectively execute the instructions for performing the data processing method.

[0335] It should be noted that the memory 136 in different computing devices 130 in the cluster of computing devices can store different instructions for performing part of the functions of the data processing method. That is, the instructions stored in the memory 136 in different computing devices 130 can implement one or more functions of the processing module.

[0336] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the data processing method.

[0337] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device and includes one or more of a memory, a floppy disk, a DVD, a Blu-ray disc, a Compact Disc Read Only Memory (CD-ROM), a Compact Disc Read-Write Memory (CD-R / W), and a semiconductor memory, etc. The computer readable storage medium includes instructions that instruct the computing device to perform the data processing method.

[0338] The embodiments of the present application further provide a chip system. The chip system includes a processor configured to implement the steps performed by the computing device cluster. In a possible design, the chip system can further include a memory configured to store necessary program instructions and data. The chip system can be composed of a chip or include the chip and other discrete devices.

[0339] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, the device and the unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0340] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0341] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0342] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0343] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A data processing method, characterized by, The method is applied to a terminal device, and the method comprises: performing feature extraction on first data to be encoded to obtain first feature data; performing context prediction on the first feature data using a first data encoding model indicated by first metadata to obtain encoding statistical information, wherein the first data encoding model is a data encoding model determined by the terminal device and a cloud server in consistency negotiation before encoding the first data, and a data quantization precision of the first data encoding model indicated by the first metadata matches a data quantization precision of a first data decoding model used by the cloud server; encoding the first feature data according to the encoding statistical information to obtain a first code stream; compressing the first metadata and the first code stream to obtain a compressed code stream, wherein the compressed code stream is used for sending to the cloud server.

2. The method of claim 1, wherein, The method further comprises: the terminal device respectively acquires resource information of the terminal device and resource information of the cloud server; performs consistency detection on whether the terminal device and the cloud server respectively support quantization precision according to the resource information of the terminal device and the resource information of the cloud server, and obtains a consistency detection result; determines the first data encoding model from a model set of the terminal device according to the consistency detection result; generates the first metadata according to the data quantization precision of the first data encoding model.

3. The method of claim 1, wherein, The generating the first metadata according to the data quantization precision of the first data encoding model comprises: determining an acceleration algorithm implementation manner of the first data encoding model according to the data quantization precision of the first data encoding model; generating the first metadata according to the data quantization precision of the first data encoding model and the acceleration algorithm implementation manner of the first data encoding model.

4. The method of claim 1, wherein, The performing consistency detection on whether the terminal device and the cloud server respectively support quantization precision according to the resource information of the terminal device and the resource information of the cloud server comprises: performing offline consistency detection according to the resource information of the terminal device and the resource information of the cloud server; or performing online consistency detection by real-time interaction with the cloud server according to the resource information of the terminal device and the resource information of the cloud server.

5. A data processing method, characterized by, The method is applied to a cloud server, and the method comprises: decoding a compressed code stream from a terminal device to obtain decoded second feature data; performing context prediction on the second feature data using a first data decoding model indicated by second metadata to obtain decoding statistical information, wherein the first data decoding model is a data decoding model determined in consistency negotiation before the cloud server acquires the compressed code stream, and a data quantization precision of the first data decoding model indicated by the second metadata matches a data quantization precision of a first data encoding model used by the terminal device; decoding third feature data from the compressed code stream according to the decoding statistical information; reconstructing the third feature data according to a reconstruction model of the cloud server to obtain reconstructed data.

6. A data processing apparatus, characterized by, for implementing the method as claimed in any one of claims 1 to 5.

7. A data processing apparatus comprising a processor and a memory coupled to the processor, the processor being configured to perform the method of any one of claims 1 to 5.

8. A computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 5.

9. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 5.

10. A chip, characterized by comprising one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from a memory of an electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device performs the method of any one of claims 1 to 5.

Images