Device-cloud collaboration and model training for deep learning model

Abstract

Embodiments of the present specification provide a device-cloud collaboration method and a model training method for a deep learning model, and a device. The deep learning model comprises a shallow network deployed on a terminal side and a non-shallow network deployed on a cloud side. The device-cloud collaboration method comprises: via the shallow network on the terminal side, performing feature extraction on content to be processed on the terminal side, so as to obtain an original content representation corresponding to the content to be processed; and performing feature compression on the original content representation, sending an obtained compressed content representation to the cloud side, and restoring, via the cloud side, the compressed content representation to a target content representation corresponding to the content to be processed.

Description

End-to-end collaboration and model training for deep learning models Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method for end-cloud collaboration for deep learning models, a method for model training of deep learning models, and corresponding electronic devices. Background Art

[0002] With the development of artificial intelligence technology, deep learning models, such as large models, are becoming more and more widely used. How to deploy large models has become a focus of attention.

[0003] In one technical solution, a deep learning model, such as a large model, is deployed in the cloud, and terminal-side content data, such as images, is transmitted to the cloud for processing by the large cloud-based model. However, this technical solution requires transmitting terminal-side content data, such as images, from the mobile device to the cloud. Because terminal-side content data may involve user privacy, it can easily lead to privacy leaks.

[0004] Therefore, how to avoid privacy leakage in the application of deep learning models has become a technical problem that needs to be solved urgently.

[0005] The content of the background technology section is merely information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the Invention

[0006] This specification provides a deep learning model end-cloud collaboration method, a deep learning model training method, and an electronic device, which can avoid privacy leakage problems in the application of deep learning models.

[0007] In the first aspect, the present specification provides a terminal-cloud collaboration method for a deep learning model, wherein the deep learning model includes a shallow network deployed on the terminal side and a non-shallow network deployed on the cloud side. The method includes: performing feature extraction on the content to be processed on the terminal side through the shallow network on the terminal side to obtain the original content representation corresponding to the content to be processed; performing feature compression on the original content representation to obtain a compressed content representation corresponding to the original content representation; and sending the compressed content representation to the cloud side to restore the compressed content representation to the target content representation through the cloud side.

[0008] In an example embodiment, based on the above scheme, an encoding network is deployed on the terminal side, which is an independent network or a part of the shallow network. The feature compression includes feature dimensionality reduction, and the feature compression of the original content representation to obtain a compressed content representation corresponding to the original content representation includes: performing feature dimensionality reduction on the original content representation through the encoding network to obtain a reduced dimensionality content representation corresponding to the original content representation.

[0009] In an example embodiment, based on the above scheme, the feature compression also includes quantization processing, and the feature compression of the original content representation to obtain a compressed content representation corresponding to the original content representation also includes: performing the quantization processing on each value of the feature vector of the reduced-dimensional content representation to obtain the compressed content representation corresponding to the original content representation.

[0010] In an example embodiment, based on the above scheme, the reduced-dimensionality content representation is a floating-point feature vector, the compressed content representation is an integer feature vector, and the quantization processing is performed on each value of the reduced-dimensionality content representation to obtain the compressed content representation corresponding to the original content representation, including: performing the quantization processing on each value of the floating-point feature vector to obtain the integer feature vector corresponding to the original content representation.

[0011] In an example embodiment, based on the above-mentioned solution, a decoding network is deployed on the cloud side, which is an independent network or a part of the non-shallow network. The compressed content representation is restored to the target content representation through the cloud side, including: feature decompression of the compressed content representation through the decoding network on the cloud side to obtain the target content representation.

[0012] In an example embodiment, based on the above-mentioned scheme, the feature decompression includes quantization restoration processing, and the feature decompression of the compressed content representation through the decoding network on the cloud side to obtain the target content representation includes: performing the quantization restoration processing on the compressed content representation through the decoding network on the cloud side to obtain the quantization restored content representation.

[0013] In an example embodiment, based on the above scheme, the decoding network includes a dimensionality-increasing sub-network, and the feature decompression also includes feature dimensionality-increasing processing. The feature decompression of the compressed content representation is performed by the decoding network on the cloud side to obtain the target content representation, and it also includes: performing the feature dimensionality-increasing processing on the quantized restored content representation by the dimensionality-increasing sub-network to obtain the target content representation.

[0014] On the second aspect, an embodiment of this specification provides another end-cloud collaboration method for a deep learning model, wherein the deep learning model includes a shallow network deployed on the terminal side and a non-shallow network deployed on the cloud side, and the method includes: receiving a compressed content representation of the content to be processed sent by the terminal side, wherein the compressed content representation is a compressed content representation obtained by the shallow network on the terminal side performing feature extraction and feature compression on the content to be processed; and restoring the compressed content representation to a target content representation through the cloud side.

[0015] On the third aspect, an embodiment of the present specification provides a model training method for a deep learning model, wherein the deep learning model includes a shallow network deployed on the terminal side and a non-shallow network deployed on the cloud side, and the method includes: performing feature extraction on the sample content through the shallow network on the terminal side to obtain the original sample content representation corresponding to the sample content; performing feature compression on the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation; sending the compressed sample content representation to the cloud side to restore the compressed sample content representation to the target sample content representation through the cloud side; determining the model loss of the deep learning model based on the difference between the target sample content representation and the original sample content representation, so as to adjust the model parameters of the deep learning model based on the model loss.

[0016] In an example embodiment, based on the above scheme, an encoding network is deployed on the terminal side, and the encoding network is an independent network or a part of the shallow network. The feature compression includes feature dimensionality reduction, and the feature compression of the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation includes: performing the feature dimensionality reduction on the original sample content representation through the encoding network to obtain a reduced dimensionality content representation corresponding to the original sample content representation.

[0017] In an example embodiment, based on the above scheme, the feature compression also includes a cropping process, wherein the feature compression of the original sample content representation is performed to obtain a compressed sample content representation corresponding to the original sample content representation, and further includes: performing the cropping process on each value of the feature vector of the reduced-dimensional content representation to obtain a cropped content representation corresponding to the original sample content representation.

[0018] In an example embodiment, based on the above scheme, the feature compression also includes quantization processing, and the feature compression of the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation also includes: determining a quantization scale factor corresponding to the reduced-dimensional content representation; based on the quantization scale factor, quantizing each value of the feature vector of the reduced-dimensional content representation to obtain the compressed sample content representation.

[0019] In an example embodiment, based on the above scheme, a decoding network is deployed on the cloud side, which is an independent network or a part of the non-shallow network. The compressed sample content representation is restored to the target sample content representation through the cloud side, including: feature decompression of the compressed sample content representation through the decoding network on the cloud side to obtain the target sample content representation.

[0020] In an example embodiment, based on the above scheme, the feature decompression includes quantization restoration processing, and the feature decompression of the compressed sample content representation through the decoding network on the cloud side to obtain the target sample content representation includes: performing the quantization restoration processing on the compressed content representation through the decoding network on the cloud side to obtain a first quantization restoration content representation; multiplying each value of the feature vector of the quantization restoration content representation by the quantization scale factor to obtain a second quantization restoration content representation.

[0021] In an example embodiment, based on the above scheme, the decoding network includes a dimensionality-increasing sub-network, and the feature decompression also includes feature dimensionality-increasing processing. The feature decompression of the compressed sample content representation is performed by the decoding network on the cloud side to obtain the target sample content representation. It also includes: performing the feature dimensionality-increasing processing on the second quantized restored content representation by the dimensionality-increasing sub-network to obtain the target sample content representation.

[0022] In an exemplary embodiment, based on the above solution, the model loss includes feature difference and feature distance.

[0023] In an example embodiment, based on the above scheme, the model loss of the deep learning model is determined based on the difference between the target sample content representation and the original sample content representation, including: determining the feature difference and the feature distance between the feature vector of the target sample content representation and the feature vector of the original sample content representation; performing a weighted operation on the feature difference and the feature distance to obtain the model loss of the deep learning model.

[0024] In an exemplary embodiment, based on the above solution, the characteristic difference is an absolute value difference, and the characteristic distance is a cosine distance.

[0025] In a fourth aspect, this specification also provides an electronic device comprising: at least one storage medium storing at least one instruction set for end-cloud collaborative processing; and at least one processor communicatively connected to the at least one storage medium, wherein, when the electronic device is running, the at least one processor reads the at least one instruction set and executes the end-cloud collaborative method of the deep learning model described in the first aspect of this specification according to the instructions of the at least one instruction set.

[0026] In the fifth aspect, this specification also provides an electronic device, comprising: at least one storage medium storing at least one instruction set for model training processing; and at least one processor communicatively connected to the at least one storage medium, wherein when the device is running, the at least one processor reads the at least one instruction set and executes the model training method described in the third aspect of this specification according to the instructions of the at least one instruction set.

[0027] It can be seen from the above technical solutions that the end-cloud collaboration method of the deep learning model, the model training method of the deep learning model, and the electronic device provided in the embodiments of this specification, on the one hand, deploy the shallow network of the deep learning model on the terminal side and deploy the non-shallow network of the deep learning model on the cloud side, and extract features of the content to be processed on the terminal side through the shallow network on the terminal side to obtain the original content representation corresponding to the content to be processed, and can extract the representation data of the content to be processed on the terminal side through a deep learning model, such as a large model, to improve the recognition ability of the model on the terminal side; on the other hand, the original content representation is feature compressed to obtain a compressed content representation corresponding to the original content representation, and the compressed content representation after feature compression is sent to the cloud side, which can reduce the amount of data transmitted on the terminal side; on the other hand, the compressed content representation is restored to the target content representation through the cloud side. Since only the intermediate layer embedded features of the content to be processed on the terminal side are transmitted to the cloud, the original content is avoided from being transmitted to the cloud, thereby avoiding privacy leakage problems in the application of deep learning models.

[0028] Other features of the end-cloud collaboration method and device for deep learning models provided in this specification are partially listed in the following description. Based on the description, the following figures and examples will be apparent to those of ordinary skill in the art. The creative aspects of the end-cloud collaboration method and device for deep learning models provided in this specification can be fully explained through practice or use of the methods, devices, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 shows a schematic diagram of an implementation environment of an end-cloud collaboration method for a deep learning model provided in an embodiment of this specification.

[0030] FIG2 shows a hardware structure diagram of an electronic device 200 provided according to an embodiment of this specification.

[0031] FIG3 shows a flow chart of a method for end-cloud collaboration of a deep learning model provided according to some embodiments of this specification.

[0032] FIG4 shows an architectural diagram of an implementation architecture of a terminal-cloud collaboration method provided according to some embodiments of this specification.

[0033] FIG5 shows a flow chart of a method for end-cloud collaboration of providing a deep learning model according to other embodiments of this specification.

[0034] FIG6 shows a flow chart of a method for end-cloud collaboration of providing a deep learning model according to some further embodiments of this specification.

[0035] FIG7 shows a flow chart of a model training method for a deep learning model provided according to some embodiments of this specification. DETAILED DESCRIPTION

[0036] The following description provides specific application scenarios and requirements for this specification, with the goal of enabling those skilled in the art to make and use the contents of this specification. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but is intended to be accorded the broadest scope consistent with the claims.

[0037] The terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. For example, as used herein, the singular forms "a," "an," and "the" may also include the plural forms unless the context clearly indicates otherwise. When used in this specification, the terms "comprise," "include," and / or "contain" are intended to refer to the presence of the associated integers, steps, operations, elements, and / or components, but do not preclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups or the addition of other features, integers, steps, operations, elements, components, and / or groups in the system / method.

[0038] These and other features of this specification, as well as the operation and function of the associated elements of the structure, and the economical assembly and manufacture of the components, can be significantly improved with consideration of the following description. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0039] The flowcharts used in this specification illustrate operations implemented by systems according to some embodiments of the present specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. Rather, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0040] First, the terms involved in one or more embodiments of this specification are explained.

[0041] Large models: These refer to deep learning models with large parameters and complex computational structures. These models are typically constructed from deep neural networks, which contain multiple layers of neural networks and billions or even hundreds of billions of parameters.

[0042] Shallow network: It is a layer or several layers of network close to the input layer in the deep neural network of the large model. The non-shallow network is the network structure outside the shallow network of the large model.

[0043] Feature compression: In machine learning or deep learning, feature compression involves processing high-dimensional feature vectors or matrices of data to reduce storage space requirements and computational complexity while maintaining the validity of the original information. Feature compression can include methods such as dimensionality reduction, quantization, and encoding.

[0044] Feature decompression: During model processing or data preprocessing, if the content representation of the original content is compressed in some form, such as quantization or dimensionality reduction, then "feature decompression" can be understood as the process of restoring these compressed content representations to the original, uncompressed content representations.

[0045] Based on the above content, the embodiments of this specification provide a method for end-cloud collaboration of a deep learning model, a model training method of a deep learning model, and an electronic device. On the one hand, the shallow network of the deep learning model is deployed on the terminal side and the non-shallow network of the deep learning model is deployed on the cloud side. The shallow network on the terminal side performs feature extraction on the content to be processed on the terminal side to obtain the original content representation corresponding to the content to be processed. The representation data of the content to be processed on the terminal side can be extracted through a deep learning model, such as a large model, to improve the recognition ability of the model on the terminal side; on the other hand, the feature compression of the original content representation is performed to obtain a compressed content representation corresponding to the original content representation, and the compressed content representation after feature compression is sent to the cloud side, which can reduce the amount of data transmitted on the terminal side; on the other hand, the compressed content representation is restored to the target content representation through the cloud side. Since only the intermediate layer embedded features of the content to be processed on the terminal side are transmitted to the cloud, the original content is avoided from being transmitted to the cloud, thereby avoiding privacy leakage problems in the application of deep learning models.

[0046] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0047] FIG1 shows a schematic diagram of an implementation environment of an end-cloud collaboration method for a deep learning model provided in an embodiment of this specification.

[0048] As shown in FIG. 1 , the implementation environment 100 may include a terminal 110 , a server 130 , and a database 140 .

[0049] The terminal 110 is connected to the server 130 via a wireless network or a wired network 120. The terminal 110 may be a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto.

[0050] The terminal 110 may store data or instructions for executing the end-cloud collaboration method for the deep learning model described in this specification. The terminal 110 may include a hardware device with data information processing capabilities and the necessary programs required to drive the hardware device to work.

[0051] Server 130 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 130 provides background services for applications running on terminal 110.

[0052] The server 130 is equipped with an integrated development platform. The integrated development platform, also known as the integrated development environment (IDE), is an application used to provide a program development environment, generally including tools such as a code editor, a compiler, a debugger, and a graphical user interface. Developers can write program code (i.e., program development) on the integrated development platform. The integrated development platform server can be a computing device specifically used by the integrated development platform to implement the end-cloud collaboration method of the deep learning model. The server 130 can communicate data with the terminal 110 and the database 140 respectively.

[0053] In addition, the server 130 may store data or instructions for executing the end-cloud collaborative method of the deep learning model described in this specification. The server 130 may include a hardware device with data information processing capabilities and the necessary programs to drive the hardware device to work. Of course, the server 130 may also be only a hardware device with data processing capabilities, or only a program running in the hardware device. In some embodiments, the server 130 may also be a plug-in and deployed on the terminal 110. In this case, the server 130 stores data or instructions for executing the end-cloud collaborative method of the deep learning model corresponding to the terminal 110 described in this specification.

[0054] The database 140 can store data and / or instructions. In some embodiments, the database 140 can store content to be processed and sample content. In some embodiments, the database 140 can store data and / or instructions for the end-cloud collaborative method of the deep learning model described in this specification executed by the server 130 or used to execute it. The terminal 110 and the server 130 have access to the database 140, and the terminal 110 and the server 130 can access the data or instructions stored in the database 140 through the network. In some embodiments, the database 140 can be directly connected to the terminal 110 and the server 130. In some embodiments, the database 140 can be part of the server 130. In some embodiments, the database 140 may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM) or the like, or any combination thereof. Exemplary mass storage may include non-transitory storage media such as disks, optical disks, solid-state drives, etc. Example removable storage may include flash drives, floppy disks, optical disks, memory cards, zip disks, tapes, etc. Typical volatile read-write memory may include random access memory (RAM). Example RAMs may include dynamic RAM (DRAM), double date rate synchronous dynamic RAM (DDRSDRAM), static RAM (SRAM), thyristor RAM (T-RAM), zero capacitance RAM (Z-RAM), etc. Example ROMs may include mask ROM (MROM), programmable ROM (PROM), virtually programmable ROM (PEROM), electronically programmable ROM (EEPROM), compact disc ROM (CD ROM), digital versatile disk ROM, etc.

[0055] Those skilled in the art will appreciate that the number of terminals can be greater or lesser. For example, there may be only one terminal, or there may be dozens, hundreds, or even more terminals, in which case the implementation environment may also include other terminals. The embodiments of this specification do not limit the number of terminals or device types.

[0056] After introducing the implementation environment of the embodiments of this specification, the application scenarios of the embodiments of this specification will be introduced in combination with the above implementation environment. In the following description, the terminal is the terminal 110 in the above implementation environment, and the server is the server 130 in the above implementation environment. The technical solutions provided by the embodiments of this specification can be applied in scenarios of deep learning applications or large model applications, such as large image model applications, large text model applications, and large audio model applications.

[0057] Taking the application of the technical solution provided in the embodiments of this specification in the application of a large image model as an example, the large image model includes a shallow network deployed on the terminal side, such as the terminal 110, and a non-shallow network deployed on the cloud side, such as the server 130. The content to be processed may be an image to be processed. The terminal 110 performs feature extraction on the image to be processed on the terminal side through the shallow network on the terminal side to obtain the original content representation corresponding to the image to be processed; performs feature compression on the original content representation to obtain a compressed content representation corresponding to the original content representation; and sends the compressed content representation to the cloud side, such as the server 130, so as to restore the compressed content representation to the target content representation through the cloud side.

[0058] It should be noted that the above is explained using the application of the technical solution provided in the embodiments of this specification in image large model applications as an example. The technical solution provided in the embodiments of this specification can also be applied to other appropriate large model applications such as multimodal large model applications, video large model applications, etc. The implementation process belongs to the same inventive concept as the above description and will not be repeated here.

[0059] It should be noted that the steps in the end-cloud collaboration method of the deep learning model in the example embodiment of this specification can be partially executed by the client, partially executed by the server, or all executed by the server or all executed by the client. This specification does not specifically limit this.

[0060] Based on the implementation environment shown in Figure 1, the end-cloud collaboration method and electronic device for the deep learning model provided by the embodiments of this specification will be described in detail below in conjunction with Figures 2-7. It should be noted that the above implementation environment is only shown to facilitate understanding of the spirit and principles of the embodiments of this specification, and the embodiments of this specification are not limited in this respect. On the contrary, the embodiments of this specification can be applied to any applicable scenario.

[0061] Figure 2 is a structural diagram of an electronic device 200 provided according to some embodiments of this specification. The above-mentioned electronic device 200 can execute the end-cloud collaboration method or model training method of the deep learning model described in this specification. The end-cloud collaboration method or model training method of the deep learning model is introduced in other parts of this specification. The above-mentioned electronic device 200 can be a general-purpose computer or a special-purpose computer. For example, the above-mentioned electronic device 200 can be a server, a personal computer, a portable computer (such as a notebook computer, a tablet computer, etc.), or other electronic devices with computing capabilities. Of course, the above-mentioned electronic device can be the terminal 110 or the server 130 in Figure 1, or it can be a terminal device used by multiple developers to develop programs on an integrated development platform.

[0062] The electronic device in this specification may include one or more of the following components: a processor 210 , a memory 220 , an input device 230 , an output device 240 , and a bus 250 . The processor 210 , the memory 220 , the input device 230 , and the output device 240 may be connected via the bus 250 .

[0063] The processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect the various parts of the entire electronic device, and executes the end-cloud collaboration method or model training method of the deep learning model described in this specification by running or executing instructions, programs, code sets or instruction sets stored in the memory 220, and calling data stored in the memory 220. Optionally, the processor 210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 210 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to handle wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 210 and may be implemented separately through a communication chip.

[0064] The memory 220 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 220 includes a non-transitory computer-readable storage medium. The memory 220 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system, including a system deeply developed based on the IOS system, or other systems.

[0065] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0066] The input device 230 is used to receive input commands or data and includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch-sensitive device. The output device 240 is used to output commands or data and includes, but is not limited to, a display device and a speaker. In one example, the input device 230 and the output device 240 may be combined, and the input device 230 and the output device 240 may be a touch-sensitive display.

[0067] In addition, those skilled in the art will understand that the structures of the electronic devices shown in the above figures do not limit the electronic devices. The electronic devices may include more or fewer components than shown, or may combine certain components, or arrange the components differently. For example, the electronic devices may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, and other components, which will not be described in detail here.

[0068] Figure 3 shows a flow chart of a method for end-cloud collaboration of a deep learning model provided according to an embodiment of this specification. As before, the electronic device 200 can execute the method for end-cloud collaboration of a deep learning model of an embodiment of this specification. Specifically, the processor 210 can read the instruction set stored in its local storage medium, and then execute the method for end-cloud collaboration of a deep learning model of an embodiment of this specification according to the provisions of the instruction set. Below, steps S310 to S330 in the method for end-cloud collaboration of a deep learning model will be described in detail with reference to the accompanying drawings.

[0069] As shown in FIG3 , in step S310 , feature extraction is performed on the content to be processed on the terminal side through a shallow network on the terminal side to obtain an original content representation corresponding to the content to be processed.

[0070] In an example embodiment, the deep learning model includes a shallow network deployed on the terminal side and a non-shallow network deployed on the cloud side. The deep learning model can be a large image model, a large text model, or a large audio model. For example, the large model can be a large image model based on the Transformer architecture. The terminal side can be a smartphone terminal, an IoT terminal, a laptop computer, or a desktop computer. The content to be processed on the terminal side can be data in various forms, such as images, text, or audio.

[0071] A large model refers to a deep neural network with a large number of overall parameters and a complex structure. The shallow network of a large model refers to the portion of the network structure that is closer to the input layer and has a relatively small hierarchy. For example, a shallow network is one or several shallow layers close to the input layer of the deep neural network of a large model. For example, in a deep neural network of a large model, if the deep neural network is divided into an input layer, a hidden layer, and an output layer, where the hidden layer includes shallow networks and deep networks (i.e., non-shallow networks), then the shallow network of the large model refers to the portion of the network structure that has one or a limited number of hidden layers close to the input layer. Compared to deep networks, shallow networks have lower computational complexity and consume fewer resources.

[0072] Furthermore, the terminal side is an electronic device 200, and the electronic device 200 performs feature extraction on the content to be processed on the terminal side through the shallow network on the terminal side, and obtains the original content representation corresponding to the content to be processed. For example, assuming that the content to be processed is an image, the image to be processed is input into the first convolution layer of the shallow network on the terminal side, and multiple convolution kernels scan and calculate the convolution results of each local area of ​​the image with the weight matrix, generating multiple feature maps (feature map), and obtaining the original image representation corresponding to the image to be processed. With each layer of convolution operation, the spatial resolution of the feature will gradually decrease, and the depth of the feature (ie, the number of channels) will increase, and the represented information will become more and more abstract and have stronger recognition capabilities. A shallow network is used on the terminal side to perform low-level to high-level feature extraction on the image to be processed, and a feature vector that can better reflect the characteristics of the original content is generated, which is convenient for further processing and analysis downstream.

[0073] It should be noted that although the deep learning model is described as a large image model, ordinary technicians in this field should understand that the deep learning model can also be other appropriate deep learning models such as a video deep learning model or a multimodal deep learning model, which is also within the scope of the embodiments of this specification.

[0074] Taking the image to be processed as an example, as shown in Figure 4, the input size of the image to be processed, or the original image, is [3, 224, 224] in RGB format, i.e., three channels, with each channel being 224x224 pixels. The electronic device 200 performs feature extraction on the image using a convolutional neural network layer, generating a floating-point feature vector (embedding) of dimension [1280, 16, 16], i.e., a representation of the original content. This convolutional neural network layer contains multiple convolution kernels, each of which slides and calculates on the features of the previous layer to generate the original content representation corresponding to the image to be processed, i.e., the intermediate-layer embedding vector. In other words, this convolutional neural network layer applies 1280 independent convolution kernels (filters) to the image to be processed. Each convolution kernel generates a feature map or feature graph, which together constitute the output of this layer. Each channel of the output can be understood as a specific type of feature or pattern extracted from the input data. The 1280 channels represent the model's ability to simultaneously learn and represent 1280 different feature combinations.

[0075] Furthermore, since the parameters of the floating-point feature vector are of floating-point float32 type, and the float32 type contains 4 bytes, the output vector size of this layer of the network is 1280KB (1280*16*16*4 / 1024). If the network layer is placed directly on the mobile terminal, each original image will obtain an output vector result of 1280KB to be transmitted back to the cloud. At this time, when the image level on the mobile terminal is large, it will consume a huge amount of storage and transmission. Therefore, it is necessary to perform feature compression processing on the original content representation of the image. In step S320, the original content representation is feature compressed to obtain a compressed content representation corresponding to the original content representation.

[0076] In an example embodiment, feature compression refers to the processing of high-dimensional feature vectors or feature matrices of data in machine learning or deep learning to reduce its storage space requirements and computational complexity while maintaining the validity of the original information as much as possible. Feature compression may include methods such as dimensionality reduction, quantization, and encoding. The electronic device 200 performs feature compression on the original content representation to obtain a compressed content representation corresponding to the original content representation. For example, the electronic device 200 performs feature compression on the original content representation through a coding network deployed on the terminal side to obtain a compressed content representation corresponding to the original content representation. The coding network may be an independent network, such as an encoder in an autoencoder, or it may be a network structure that is part of a shallow network of a deep learning model.

[0077] The feature compression process is described in detail below with reference to several exemplary embodiments.

[0078] In some example embodiments, the encoding network includes a dimensionality reduction subnetwork, feature compression includes feature dimensionality reduction, and the electronic device 200 performs feature dimensionality reduction on the original content representation through the dimensionality reduction subnetwork to obtain a reduced dimensionality content representation corresponding to the original content representation.

[0079] In other example embodiments, an encoding network is deployed on the terminal side, such as an encoder in an autoencoder AutoEncoder. The autoencoder includes an encoder-decoder structure. The network layer of the encoder is deployed on a mobile terminal, such as the terminal 110, and the network layer of the decoder is deployed on the cloud, such as the server 130. The electronic device 200 obtains the original content representation of the content to be processed. The original content representation is a high-dimensional feature map obtained after the content to be processed is subjected to one or more layers of convolution operations. The high-dimensional original content representation is mapped to a compact low-dimensional representation space through an encoding network, such as an encoder, to obtain a reduced-dimensional content representation corresponding to the original content representation. Referring to Figure 4, the original content representation is subjected to feature dimensionality reduction through the network layer of the encoder in the autoencoder. After the original content representation with a dimension of [1280, 16, 16] is subjected to feature dimensionality reduction, a reduced-dimensional content representation of [80, 16, 16] with a parameter type of float32 is obtained.

[0080] It should be noted that although the above-mentioned network layer dimension parameters are 1280 and 80 as examples, ordinary technicians in this field should understand that the various parameters in the embodiments of this specification are variable, such as the dimension of the input data, the network layer dimension, the vector clipping range and other parameters are all variable. For example, the network layer dimension parameter can be 1360 or 90, etc., and the embodiments of this specification do not specifically limit this.

[0081] In other example embodiments, feature compression also includes quantization processing, and the electronic device 200 quantizes each value of the feature vector of the reduced-dimensional content representation to obtain a compressed content representation corresponding to the original content representation. Quantization is the discretization of continuous feature values, converting them into a finite number of levels or intervals. For example, a floating-point feature of type float32 can be converted into an int8 or other lower-precision data type to save storage space. For example, the reduced-dimensional content representation is a floating-point feature vector, and the compressed content representation is an integer feature vector. The electronic device 200 quantizes each value of the floating-point feature vector, such as a float32-type feature vector, to obtain an integer feature vector, such as an int8-type feature vector, corresponding to the original content representation. As shown in Figure 4, after the floating-point feature vector (embedding) with a dimension of [1280, 16, 16], that is, the 1280kb original content representation, is subjected to feature dimensionality reduction, a reduced-dimensional content representation of [80, 16, 16] with a parameter type of float32 is obtained. After the reduced-dimensional content representation is quantized, an integer feature vector of [80, 16, 16] with a parameter type of int8 is obtained, for example, a feature vector of 20480 int8 values, that is, a compressed content representation of 20kb.

[0082] It should be noted that although feature compression including feature dimensionality reduction and quantization processing is described as an example, ordinary technicians in this field should understand that feature compression can also include other appropriate feature processing, such as feature selection or hashing, which is also within the scope of the present embodiment.

[0083] In step S330 , the compressed content representation is sent to the cloud side, so that the cloud side can restore the compressed content representation to the target content representation.

[0084] In an exemplary embodiment, the non-shallow network on the cloud side refers to a network structure other than the shallow network of a deep learning model, such as a large model. A shallow network is a network structure other than the shallow network of a large model's deep neural network that is shallower than the input layer, and a non-shallow network is a network structure other than the shallow network of the large model. For example, in a large model's deep neural network, if the deep neural network is divided into an input layer, a hidden layer, and an output layer, wherein the hidden layer includes a shallow network and a deep network, i.e., a non-shallow network, then the shallow network of the large model refers to a partial network structure of one or a limited number of hidden layers close to the input layer, and the non-shallow network is a hidden layer other than the shallow network.

[0085] The terminal 110, such as the electronic device 200, sends the compressed content representation corresponding to the content to be processed to the cloud side, so that the compressed content representation is restored to the target content representation through the decoding network on the cloud side. Among them, the decoding network can be an independent network, such as a decoder in an autoencoder, or it can be a network structure that is part of the non-shallow network of a deep learning model. The cloud side, such as the server 130, performs feature decompression on the compressed content representation through the decoding network on the cloud side to obtain the target content representation. The target content representation can be input into the non-shallow network of the large model for further processing for downstream use, such as classification or retrieval. During the model processing or data preprocessing process, if the content representation of the original content is compressed in some form, such as quantization, dimensionality reduction, etc., then "feature decompression" can be understood as the process of restoring these compressed content representations to the original, uncompressed content representation.

[0086] For example, a decoder of an autoencoder is deployed on the cloud side. The autoencoder includes an encoder-decoder network structure. The network layer of the encoder is deployed on a mobile terminal, such as terminal 110, and the network layer of the decoder of the encoder-decoder structure is deployed on the cloud side, such as server 130. Terminal 110, such as electronic device 200, sends the compressed content representation corresponding to the content to be processed to the cloud side, so that the decoder on the cloud side can restore the compressed content representation to the target content representation.

[0087] The feature decompression process is described in detail below with reference to several exemplary embodiments.

[0088] In some example embodiments, feature decompression includes quantization and restoration processing, and the cloud, such as server 130, performs quantization and restoration processing on the compressed content representation through a decoding network on the cloud side to obtain a quantized and restored content representation. For example, the compressed content representation corresponding to the original content representation is an integer feature vector, and the quantized and restored content representation is a floating-point feature vector. The electronic device 200 performs quantization and restoration processing on each value of the integer feature vector corresponding to the original content representation, such as an int8 type feature vector, to obtain a floating-point feature vector, such as a float32 type feature vector. For example, the compressed content representation of [80, 16, 16] with a parameter type of int8 is quantized and restored to a quantized and restored content representation embedding of [80, 16, 16] with a parameter type of float32.

[0089] In other example embodiments, the decoding network includes a dimensionality-raising subnetwork, and the feature decompression further includes feature dimensionality-raising processing. The cloud, such as the server 130, performs feature dimensionality-raising processing on the quantized restored content representation through the dimensionality-raising subnetwork to obtain the target content representation. For example, assuming that the feature compression adopts an autoencoder, the dimensionality-raising subnetwork is the network layer of the decoder in the autoencoder, and the cloud, such as the server 130, performs feature dimensionality-raising processing on the quantized restored content representation through the network layer of the decoder of the autoencoder to obtain a target content representation similar to the original content representation. As shown in Figure 4, the compressed content representation transmitted by the mobile terminal is quantized and restored and dimensionality-raised to obtain a target content representation embedding of [1280, 16, 16] with a parameter type of float32.

[0090] According to the technical solution in the example embodiment of Figure 3, on the one hand, the shallow network of the deep learning model is deployed on the terminal side and the non-shallow network of the deep learning model is deployed on the cloud side. The shallow network on the terminal side is used to extract features of the content to be processed on the terminal side to obtain the original content representation corresponding to the content to be processed. The deep learning model, such as a large model, can be used to extract the representation data of the content to be processed on the terminal side, thereby improving the recognition ability of the model on the terminal side. On the other hand, the original content representation is feature compressed to obtain a compressed content representation corresponding to the original content representation, thereby reducing the amount of data transmitted on the terminal side. On the other hand, the compressed content representation after feature compression is sent to the cloud side, and the compressed content representation is restored to the target content representation by the cloud side. Since only the intermediate layer embedded features of the content to be processed on the terminal side are transmitted to the cloud, the original content is avoided from being transmitted to the cloud, thereby avoiding privacy leakage problems in the application of deep learning models.

[0091] FIG5 shows a flow chart of a method for end-cloud collaboration of providing a deep learning model according to other embodiments of this specification.

[0092] As shown in FIG5 , in step S510 , features of the original image are extracted through a shallow network on the terminal side to obtain a representation of the original content.

[0093] In an exemplary embodiment, the electronic device 200 extracts features from the original image on the terminal side through a shallow network on the terminal side to obtain the original content representation corresponding to the original image. The shallow network is a shallow layer or layers of networks close to the input layer in the deep neural network of the large model. As shown in FIG6 , assuming that the input size of the original image is an RGB format of [3,224,224], i.e., 3 channels, each channel 224x224 pixels, the electronic device 200 extracts features from the image to be processed through a layer of convolutional neural network to obtain a floating-point feature vector (embedding) of dimension [1280,16,16], i.e., the original content representation. This layer of convolutional neural network contains multiple convolution kernels, each of which slides and calculates on the features of the previous layer, thereby generating the original content representation corresponding to the image to be processed, i.e., the intermediate layer embedding vector.

[0094] In other words, this convolutional neural network layer applies 1,280 independent convolution kernels (filters) to the processed image. Each convolution kernel generates a feature map, or feature graph, which together form the output of the layer. Each channel of the output can be understood as a specific type of feature or pattern extracted from the input data, and the 1,280 channels represent the model's ability to simultaneously learn and represent 1,280 different feature combinations.

[0095] In step S520 , dimensionality reduction processing is performed on the original content representation corresponding to the original image to obtain a reduced-dimensional content representation.

[0096] In an example embodiment, electronic device 200 obtains an original content representation of an original image. The original content representation is a high-dimensional feature map obtained by subjecting the original image to one or more layers of convolution operations. An encoding network, such as an encoder, maps the high-dimensional original content representation to a compact low-dimensional representation space to obtain a reduced-dimensional content representation corresponding to the original content representation. The encoding network can be an independent network, such as an encoder in an autoencoder, or it can be part of a shallow network structure in a deep learning model.

[0097] 4 and 6 , the original content representation is subjected to feature dimensionality reduction through the network layer of the encoder in the autoencoder. After the original content representation with a dimension of [1280, 16, 16] is subjected to feature dimensionality reduction, a reduced-dimensional content representation of [80, 16, 16] with a parameter type of float32 is obtained.

[0098] In step S530 , the dimensionality-reduced content representation is quantized to obtain a compressed content representation corresponding to the original image.

[0099] In an exemplary embodiment, electronic device 200 quantizes the values ​​of the feature vector of the reduced-dimensionality content representation to obtain a compressed content representation corresponding to the original content representation, and transmits the compressed content representation to the cloud. Referring to Figures 4 and 6 , after performing feature dimensionality reduction on a floating-point feature vector (embedding) of dimension [1280, 16, 16], i.e., a 1280kb original content representation, a reduced-dimensionality content representation of [80, 16, 16] with a parameter type of float32 is obtained. After quantization of the reduced-dimensional content representation, an integer feature vector of parameter type [80, 16, 16] with an int8 parameter type is obtained, e.g., a feature vector of 20,480 int8 values, i.e., a 20kb compressed content representation.

[0100] Furthermore, in an example embodiment, referring to FIG6 , the model partially deployed on the terminal side has a size of 840 KB, the input is the original image, the input size of the original image is [3, 224, 224], and the output is 20480 vector results of int8 type (range 0-255) (the vector size is 20 KB, the calculation formula is: 20480 / 1024=20 KB).

[0101] In step S540 , the compressed content representation is quantized and restored to obtain a quantized and restored content representation corresponding to the original image.

[0102] In an example embodiment, the cloud, such as the server 130, receives the model result uploaded by the terminal side, i.e., the compressed content representation, and performs quantization and restoration processing on the compressed content representation through the decoding network on the cloud side to obtain the quantized and restored content representation. Among them, the decoding network can be an independent network, such as a decoder in an autoencoder, or it can be a network structure that is part of a non-shallow network of a deep learning model. For example, the compressed content representation corresponding to the original content representation is an integer feature vector, and the quantized and restored content representation is a floating-point feature vector. The electronic device 200 performs quantization and restoration processing on each value of the integer feature vector corresponding to the original content representation, such as an int8 type feature vector, to obtain a floating-point feature vector, such as a float32 type feature vector. For example, the compressed content representation of [80, 16, 16] with a parameter type of int8 is quantized and restored to a quantized and restored content representation embedding of [80, 16, 16] with a parameter type of float32.

[0103] In step S550 , feature dimension upgrading is performed on the quantized restored content representation to obtain a target content representation corresponding to the original image.

[0104] In an exemplary embodiment, the cloud, such as server 130, performs feature dimensionality upscaling on the quantized and restored content representation through a dimensionality upscaling sub-network to obtain a target content representation. For example, assuming that feature compression uses an autoencoder, and the dimensionality upscaling sub-network is the network layer of the decoder in the autoencoder, the cloud, such as server 130, performs feature dimensionality upscaling on the quantized and restored content representation through the network layer of the decoder in the autoencoder to obtain a target content representation similar to the original content representation. As shown in Figures 4 and 6, the compressed content representation transmitted by the mobile terminal is quantized and restored and dimensionality upscaling is performed to obtain a target content representation embedding of [1280, 16, 16] with a parameter type of float32.

[0105] Furthermore, the target content representation corresponding to the original image can be input into a subsequent N-layer network to obtain the image representation output of the large model, which can be used downstream for classification or retrieval. Referring to Figure 6, in an example embodiment, the 20KB result obtained by the end model is transmitted back to the cloud. In the cloud, the input is 20,480 int8 vectors, which, after passing through the multi-layer deep network of the large model, ultimately output 768 float32 vectors for downstream use.

[0106] According to the technical solution in the example embodiment of Figure 5, on the one hand, the shallow network portion of the large model is deployed on the mobile device, while the deep network portion is deployed in the cloud, thereby constructing a collaborative end-to-end large model. This can address the privacy leakage issue of raw data transmission. Furthermore, feature compression methods such as dimensionality reduction and quantization are used to compress the output results of the shallow network on the mobile device. For example, the original 1280KB of data can be converted into 20KB / 40KB of data, reducing the amount of data transmitted. Furthermore, after the target content representation is obtained in the cloud through dimensionality increase and restoration, which is close to the original content representation, the performance loss of downstream tasks can be controlled to a small range, for example, within 2%.

[0107] Figure 7 shows a flow chart of a model training method for a deep learning model provided according to some embodiments of this specification. As before, the electronic device 200 can execute the model training method of the embodiment of this specification. Specifically, the processor 210 can read the instruction set stored in its local storage medium, and then execute the model training method of the embodiment of this specification according to the provisions of the instruction set. Below, steps S710 to S740 in the model training method of the deep learning model will be described in detail with reference to the accompanying drawings.

[0108] As shown in FIG. 7 , in step S710 , feature extraction is performed on the sample content through a shallow network on the terminal side to obtain an original sample content representation corresponding to the sample content.

[0109] In an example embodiment, the deep learning model includes a shallow network deployed on the terminal side and a non-shallow network deployed on the cloud side. The deep learning model can be a large image model, a large text model, or a large audio model. For example, the large model can be a large image model based on the Transformer architecture. The terminal side can be a smartphone terminal, an IoT terminal, a laptop computer, or a desktop computer. The sample content can be data in various forms, such as images, text, or audio data. For example, multiple images can be randomly selected as sample images.

[0110] The terminal, such as the electronic device 200, extracts features from the sample content through a shallow network on the terminal side to obtain an original sample content representation corresponding to the sample content. The implementation process and implementation effect of step S710 are similar to the implementation process and implementation effect of step S310 above, and will not be repeated here.

[0111] In step S720 , feature compression is performed on the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation.

[0112] In an example embodiment, feature compression may include methods such as dimensionality reduction, quantization, and encoding. A terminal, such as electronic device 200, performs feature compression on the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation. For example, electronic device 200 performs feature dimensionality reduction on the original sample content representation using an encoding network deployed on the terminal side to obtain a reduced-dimensional content representation corresponding to the original sample content representation. The encoding network may be an independent network, such as an encoder in an autoencoder, or may be a network structure that is part of a shallow network of a deep learning model.

[0113] In some example embodiments, the encoding network on the terminal side includes a dimensionality reduction subnetwork, feature compression includes feature dimensionality reduction, and the electronic device 200 performs feature dimensionality reduction on the original sample content representation through the dimensionality reduction subnetwork to obtain a reduced dimensionality content representation corresponding to the original sample content representation.

[0114] In other example embodiments, feature compression also includes quantization processing, and the electronic device 200 determines a quantization scale factor corresponding to the reduced-dimensional content representation; based on the quantization scale factor, each value of the feature vector of the reduced-dimensional content representation is quantized to obtain a compressed sample content representation, wherein the quantization scale factor determines the mapping relationship between the original data and the quantized data.

[0115] For example, the electronic device 200 determines the quantization scale factor corresponding to the reduced-dimensionality content representation based on the target quantization type. For example, the quantization scale factor is determined based on the maximum absolute value of the reduced-dimensionality content representation vector, which can ensure that all floating-point numbers can be mapped to the integer value range of the target quantization interval. For example, in quantizing the reduced-dimensionality content representation from a floating-point type to an int8 type, the interval represented by the int8 type is [-128, 127], and the corresponding quantization scale factor can be obtained through calculation; next, the quantization value of each value of the eigenvector of the reduced-dimensionality content representation is calculated: each floating-point number is divided by the quantization scale factor and rounded to the nearest integer value to obtain the compressed sample content representation corresponding to the original sample content representation.

[0116] The implementation process and implementation effect of the undetailed part of step S720 are similar to the implementation process and implementation effect of the above-mentioned step S320, and will not be repeated here.

[0117] In step S730 , the compressed sample content representation is sent to the cloud side, so that the compressed sample content representation is restored to the target sample content representation by the cloud side.

[0118] In an example embodiment, a terminal such as the electronic device 200 sends the compressed sample content representation to the cloud side, so that the compressed sample content representation is restored to the target sample content representation through the cloud side. For example, the electronic device 200 performs feature decompression on the compressed sample content representation through the decoding network on the cloud side to obtain the target sample content representation. Among them, the decoding network can be an independent network such as a decoder in an autoencoder, or it can be a network structure that is part of a non-shallow network of a deep learning model. The target content representation can be used downstream, such as for classification or retrieval. During the model processing or data preprocessing process, if the content representation of the original content is compressed in some form, such as quantization, dimensionality reduction, etc., then "feature decompression" can be understood as the process of restoring these compressed content representations to the original, uncompressed content representation.

[0119] The feature decompression process is described in detail below with reference to several exemplary embodiments.

[0120] In some example embodiments, feature decompression includes quantization restoration processing, and the electronic device 200 performs quantization restoration processing on the compressed sample content representation through a decoding network on the cloud side to obtain a first quantization restoration content representation; and multiplies each value of the feature vector of the quantization restoration content representation by a quantization scale factor to obtain a second quantization restoration content representation.

[0121] In other example embodiments, the decoding network includes a dimensionality increase sub-network, and the feature decompression further includes feature dimensionality increase processing. The electronic device 200 performs feature dimensionality increase processing on the second quantized restored content representation through the dimensionality increase sub-network to obtain the target sample content representation.

[0122] The implementation process and implementation effect of the undetailed part of step S730 are similar to the implementation process and implementation effect of the above-mentioned step S330, and will not be repeated here.

[0123] In step S740 , based on the difference between the target sample content representation and the original sample content representation, a model loss of the deep learning model is determined, so as to adjust model parameters of the deep learning model based on the model loss.

[0124] In this example embodiment, model loss is used to represent the difference between the model's prediction and the expected result, that is, the difference between the target sample content representation and the original sample content representation. An optimization algorithm (such as gradient descent, Adam, SGD, etc.) is used to backpropagate the model loss to update the model parameters of the deep learning model. The goal of optimization is to minimize the model loss so that the model can generate feature representations that are closer to the target content representation when processing new samples, thereby improving the model's generalization ability and prediction accuracy.

[0125] Furthermore, the model loss includes feature difference and feature distance. Feature difference loss represents the difference in value between the feature vector representing the target sample content and the feature vector representing the original sample content, while feature distance loss measures the difference in vector direction between the feature vector representing the target sample content and the feature vector representing the original sample content. Electronic device 200 determines the feature difference and feature distance between the feature vector representing the target sample content and the feature vector representing the original sample content; and performs a weighted operation on the feature difference and feature distance to obtain the model loss of the deep learning model.

[0126] For example, assuming that the feature difference loss is the absolute value loss function, the feature distance loss is the cosine loss function (Cosine Loss), the target sample content is represented by T, and the original sample content is represented by S, then the model loss of a deep learning model such as a large model can be determined by the following formula (1): L(S,T)=a*L1_loss(S,T)+b*Cosine_loss(S,T) (1)

[0127] Where a and b are learnable parameters. L1_loss(S,T) = ∑|ST|, where |...| represents the absolute value and ∑ represents the sum. Cosine_loss(S,T) = 1-(S·T) / (||S||||T||), where "·" represents the dot product of two vectors and ||...|| represents the Euclidean norm of the vector.

[0128] According to the technical solution in the example embodiment of Figure 7, on the one hand, the shallow network of the deep learning model is deployed on the terminal side and the non-shallow network of the deep learning model is deployed on the cloud side. The shallow network on the terminal side is used to extract features of the content to be processed on the terminal side to obtain the original content representation corresponding to the content to be processed. The deep learning model can be used to extract the representation data of the content to be processed on the terminal side, thereby improving the recognition ability of the model on the terminal side. On the other hand, the feature compression of the original content representation is performed to obtain a compressed content representation corresponding to the original content representation. The compressed content representation after feature compression is sent to the cloud side, thereby reducing the amount of data transmitted on the terminal side. On the other hand, the compressed content representation is restored to the target content representation through the decoding network on the cloud side. Since only the intermediate layer embedded features of the content to be processed on the terminal side are transmitted to the cloud, the original content is avoided from being transmitted to the cloud, thereby avoiding privacy leakage problems in the application of deep learning models.

[0129] Furthermore, feature compression also includes a clipping process, in which the electronic device 200 clips the individual values ​​of the feature vector of the dimensionality reduction content representation to obtain a clipped content representation corresponding to the original sample content representation. For example, the individual values ​​of the feature vector of the obtained dimensionality reduction content representation are clipped, that is, the individual values ​​of the feature vector are limited to a predetermined interval, that is, a predetermined numerical distribution range, for example, limited to [-3,3]. In other words, the values ​​of the feature vector that exceed the predetermined numerical distribution range will be adjusted to the boundary value -3 or 3. The predetermined interval can be determined based on the numerical distribution of the individual values ​​of the feature vector. For example, the predetermined interval can be the numerical interval in which 90% of the values ​​of the feature vector are located.

[0130] According to the technical solution in the above example embodiment, by performing the clipping process on each value of the feature vector representing the dimensionality reduction content, the feature values ​​can be standardized to prevent certain extreme values ​​from being too large and affecting subsequent calculations or model training.

[0131] In addition, in an example embodiment, feature compression also includes quantization processing, and the electronic device 200 performs feature compression on the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation. It also includes: the electronic device 200 determines the quantization scale factor corresponding to the cropped content representation; based on the quantization scale factor, each value of the feature vector of the cropped content representation is quantized to obtain a compressed sample content representation, wherein the quantization scale factor determines the mapping relationship between the original data and the quantized data.

[0132] For example, the electronic device 200 determines the quantization scale factor corresponding to the cropped content representation according to the target quantization type, for example, the quantization scale factor is determined based on the maximum absolute value calculation of the reduced-dimensionality content representation vector, which can ensure that all floating-point numbers can be mapped to the integer value range of the target quantization interval. For example, in the quantization of the reduced-dimensionality content representation from floating-point type to int8 type, the interval range represented by the int8 type is [-128, 127], and the corresponding quantization scale factor can be obtained by calculation. Next, the quantization value of each value of the feature vector of the cropped content representation is calculated: each floating-point number is divided by the quantization scale factor, and rounded to the nearest integer value to obtain the compressed sample content representation corresponding to the original sample content representation.

[0133] According to the technical solutions in the above-mentioned example embodiments, on the one hand, by quantizing the internal representation, floating-point numbers that occupy a large storage space can be converted into integers that occupy a smaller space; on the other hand, by selecting a suitable quantization scale factor, important information of the original data can be retained as much as possible within a limited quantization level, thereby reducing the quantization error.

[0134] The following is a flow chart illustrating a method for training a deep learning model according to other embodiments of this specification. Model training is divided into two phases: the first phase of model training includes steps S805 to S825, and the second phase of model training includes steps S830 to S870. A first predetermined number of images, such as 500,000 images, are randomly selected as the training set; and a second predetermined number of images, such as 10,000 images not in the training set, are randomly selected as the test set.

[0135] First, the model training of the first stage is described in detail in combination with steps S805 to S825.

[0136] In step S805, feature extraction is performed on the sample image through a shallow network on the terminal side to obtain the original sample content representation corresponding to the sample image.

[0137] In an exemplary embodiment, the dimension of the sample image is [3, 224, 224], which represents an RGB color image, where the first dimension 3 indicates that there are three color channels (red, green, and blue), and the second and third dimensions (224, 224) represent the width and height of the image, respectively. The electronic device 200 extracts features from the original image through a shallow network on the terminal side to obtain the original sample content representation corresponding to the original image. After the original sample image is processed by a layer of convolutional neural network (CNN), the output dimension becomes the original sample content representation vector E1 of [1280, 16, 16]. Among them, the number of output channels is 1280, which means that this layer of convolutional network generates 1280 different feature maps or feature maps, and the convolution kernel size is 14x14. This convolution kernel slides on the input image and performs calculations to extract local features. The step size is 14, and the height and width of the output are reduced to 16. Therefore, the obtained original sample content representation vector E1 is a tensor of shape [1280, 16, 16], which contains 1280 feature maps of size 16x16.

[0138] In step S810 , dimensionality reduction processing is performed on the original sample content representation corresponding to the sample image to obtain a reduced-dimensionality content representation.

[0139] In an exemplary embodiment, the electronic device 200 performs dimensionality reduction processing on the original sample content representation corresponding to the original image through a dimensionality reduction network to obtain a reduced-dimensional content representation. For example, the original sample content representation E1 passes through a layer of convolutional neural network (with 80 output channels, a kernel size of 1, and a stride of 1) to obtain a vector E2 of dimension [80, 16, 16]. The number of output channels is 80, which means that this layer of convolutional neural network will apply 80 independent convolution kernels to the original sample content representation E1. Each kernel will generate a new feature map, and all maps are stacked to form a new multi-channel feature output. The convolution kernel size is 1×1, which means that the convolution kernel only considers the relationship between a single element and its surrounding elements in each channel; the stride is 1, which means that the convolution kernel crosses one element each time it moves, and the output is a reduced-dimensional content representation of dimension [80, 16, 16], indicating that there are 80 channels, each of which is a 16×16 spatial feature map.

[0140] In step S815 , each value of the feature vector of the dimensionality-reduced content representation is clipped to obtain a clipped content representation corresponding to the original sample content representation.

[0141] In an exemplary embodiment, electronic device 200 clips each value of the feature vector of the obtained dimensionality-reduced content representation, i.e., restricting each value of the feature vector to a predetermined interval, i.e., a predetermined numerical distribution range, for example, between [-3, 3], to obtain a clipped content representation E3 corresponding to the original sample content representation. In other words, feature vector values ​​that exceed the predetermined numerical distribution range are adjusted to the boundary values ​​of -3 or 3.

[0142] In step S820 , feature dimension upgrading is performed on the cropped content representation to obtain a target sample content representation corresponding to the sample image.

[0143] In an exemplary embodiment, the cloud, such as server 130, uses a layer of convolutional neural network (with 1280 output channels, kernel size 1, and stride 1) on the cropped content representation E3 to obtain a target sample content representation vector E4 with a dimension of [1280, 16, 16]. The number of output channels is 1280, which means that the layer applies 1280 independent convolution kernels for calculation, each kernel generates a new feature map, and all maps are combined to form a new feature output with more dimensions; the convolution kernel size is 1×1, which means that the layer also performs a linear transformation on a single element on each channel without involving spatial neighborhood information. The stride is 1, that is, the convolution kernel slides element by element on the input feature map, that is, the dimension of the target sample content representation vector is [1280, 16, 16], indicating that there are 1280 channels, and each channel is still a 16×16 spatial feature map.

[0144] In step S825 , based on the difference between the target sample content representation and the original sample content representation, a model loss of the deep learning model is determined, so as to adjust the model parameters of the deep learning model based on the model loss.

[0145] In an exemplary embodiment, the mode loss, i.e., the loss function, uses a weighted version of the absolute value loss function (L1 loss) and the cosine loss function (Cosine Loss). The electronic device 200 determines the model loss of the deep learning model, such as the large model, based on the difference between the target sample content representation E4 and the original sample content representation E1. The loss function calculation formula is the following formula (2): L(E1, E4) = a*L1_loss(E1, E4) + b*Cosine_loss(E1, E4) (2)

[0146] Among them, a and b are learnable parameters, L1_loss(E1,E4)=∑|E1-E4|, where |...| represents taking the absolute value and ∑ represents summation; Cosine_loss(E1,E4)=1-(E1·E4) / (||E1||||E4||), where “·” represents the dot product of two vectors and ||...|| represents the Euclidean norm of the vector.

[0147] Next, the second stage of model training is introduced in conjunction with steps S830 to S860.

[0148] In step S830, the model parameters of the deep learning model obtained by the first stage training are loaded.

[0149] In an exemplary embodiment, when performing the second stage of training, the electronic device 200 loads the model parameters of the deep learning model obtained by the first stage of training.

[0150] In step S835, feature extraction is performed on the sample image through a shallow network on the terminal side to obtain the original sample content representation corresponding to the sample image.

[0151] In the example embodiment, the input sample image is [3, 224, 224]. After passing through a layer of convolutional neural network (output channel is 1280, kernel size is 14, stride is 14), the original sample content representation vector F1 with a dimension of [1280, 16, 16] is obtained.

[0152] In step S840 , dimensionality reduction processing is performed on the original sample content representation corresponding to the sample image to obtain a reduced-dimensionality content representation.

[0153] In the exemplary embodiment, the original sample content representation F1 is passed through a layer of convolutional neural network (with an output channel of 80, a kernel size of 1, and a stride of 1) to obtain a reduced-dimensional content representation vector F2 with a dimension of [80, 16, 16].

[0154] In step S845 , each value of the feature vector of the dimensionality-reduced content representation is clipped to obtain a clipped content representation corresponding to the original sample content representation.

[0155] In an exemplary embodiment, the vector result in step S840 , ie, the reduced-dimensional content representation F2 , is clipped and restricted to a target numerical range, for example, [-3, 3], to obtain a clipped content representation F3 .

[0156] In step S850 , each value of the feature vector of the cropped content representation is quantized to obtain a compressed sample content representation corresponding to the original sample content representation.

[0157] In an exemplary embodiment, a quantization operation is performed on the cropped content representation F3 from a floating point type to an int8 type, and the vector result of the cropped content representation of float32 type is converted into a compressed sample content representation F4 of int8 type. The specific process of the conversion from floating point type to int8 type is as follows.

[0158] 1) Calculate the quantization scale factor (scale): Take the absolute value of each value of vector F3: x = |F3|; Take the largest value in vector x: y = max(x); Calculate the quantization scale factor: scale = y / 127.0.

[0159] 2) Quantize the floating-point type to int8 type: output vector out = round(F3 / scale), where the round() method rounds the decimal part to an integer; each value in the output vector out is clipped to the range [-128, 127] and converted to int8 type to obtain the compressed sample content representation vector F4.

[0160] In step S855 , the compressed sample content representation is quantized and restored through the decoding network on the cloud side to obtain a quantized and restored content representation.

[0161] In an example embodiment, a cloud-side decoding network performs quantization restoration on the compressed sample content representation F4, converting the compressed sample content representation vector F4 (type int8) into a quantized restored content representation vector F5 (type float32). The decoding network can be an independent network, such as a decoder in an autoencoder, or a network structure that is part of a non-shallow layer of a deep learning model. The specific quantization restoration process is as follows: the value in the compressed sample content representation vector F4 is converted to a float32 type; this value is then multiplied by a quantization scale factor to obtain the quantized restored content representation vector F5.

[0162] In step S860, the feature dimension increase processing is performed on the quantized restored content representation to obtain the target sample content representation.

[0163] In an exemplary embodiment, a convolutional neural network (with an output channel of 1280, a kernel size of 1, and a stride of 1) is used on the quantized restored content representation vector F5 of step S855 to obtain a target sample content representation vector F6 with a dimension of [1280, 16, 16].

[0164] In step S865 , based on the difference between the target sample content representation and the original sample content representation, a model loss of the deep learning model is determined to adjust model parameters of the deep learning model based on the model loss.

[0165] In the exemplary embodiment, the loss function uses a weighted version of the absolute value loss function (L1 loss) and the cosine loss function (Cosine Loss). The loss function calculation formula is as follows (3): L(F1, F6) = a*L1_loss(F1, F6) + b*Cosine_loss(F1, F6) (3)

[0166] Where a and b are learnable parameters. L1_loss(F1,F6) = ∑|F1,F6|, where |...| represents absolute value and ∑ represents summation. Cosine_loss(F1,F6) = 1-(F1·F6) / (||F1||||F6||), where "·" represents the dot product of two vectors and ||...|| represents the Euclidean norm of the vector.

[0167] In step S870, a trained deep learning model is obtained through continuous training.

[0168] In the first and second phases of training in the above example embodiment, the shallow network layer on the terminal side has its parameters frozen during training, meaning that this network layer does not participate in training. This is because the original large model used is a pre-trained model. To maintain the performance of the original large model, the parameters of the shallow network of the original large model do not need to be modified.

[0169] It should be further explained that, although the above-mentioned network layer dimension parameters are 1280 and 80 as examples, ordinary technicians in this field should understand that the various parameters in the embodiments of this specification are variable, such as the dimension of the input data, the network layer dimension, the vector clipping range and other parameters are all variable. For example, the network layer dimension parameter can be 1360 or 90, etc., and the embodiments of this specification do not specifically limit this. According to the technical solution in the above-mentioned example embodiment, on the one hand, the shallow network part of the large model is deployed on the mobile terminal, and it is proposed to use dimensionality reduction, quantization and other methods to reduce the size of the transmission volume to reduce data storage consumption; on the other hand, by using the shallow network of the large model to extract the representation data of the original content, the recognition ability of the model on the terminal side is improved; on the other hand, the shallow network of the large model is placed on the mobile terminal, and the input of the cloud part only requires the representation data embedded in the middle layer of the original content, rather than the original image, thereby solving the problem of privacy leakage.

[0170] Furthermore, in this example embodiment, a model evaluation is performed on the large model based on the target sample content representation obtained by a trained deep learning model, such as a large model. The evaluation dataset can be selected from a portion of the common datasets ImageNet, Cifar100, and COCO-CN dataset, and the following two methods are used for evaluation.

[0171] 1) Linear probing method: It is a technique used to evaluate the quality of feature representation or embedding. The basic steps are: 1) Use a trained model (such as a large pre-trained model) to generate feature embeddings; 2) Use these feature embeddings as input to train a simple linear classifier (such as logistic regression) to perform the target task; 3) Evaluate the performance of this linear classifier on the test set to measure the effectiveness of the feature representation. In the Linear probing method, after the large model outputs the embedding, it is connected to the fully connected network classification head and then trained to obtain an image classification model. The comparison results of the large model of this solution and the original large model are shown in Table 1 below. Table 1. Comparison results of the large model of this solution and the original large model obtained by the linear probing method

[0172] 2) Embedding Retrieval method: It is a search and retrieval process used to find the most relevant or similar items to a given query in an embedding space. The basic steps are: 1) Use the model to convert all items in the dataset into embedding vectors; 2) Convert the query items into corresponding embedding vectors; 3) Usually use measurement methods such as cosine similarity to calculate the similarity between the query embedding and each embedding in the dataset in the embedding space; 4) Sort by similarity score and retrieve the most relevant items. In the Embedding Retrieval method, the downstream is no longer trained, and the embedding of the image and text is used to calculate the similarity. The comparison results of the large model of this scheme and the original large model are shown in Table 2 below. Table 2. Comparison results of the large model of this scheme and the original large model obtained by the embedding retrieval method

[0173] As shown above, the technical solution of the embodiment of this specification, through the solution of dimensionality reduction and restoration + quantization, compares the results of the original large model: (1) If the output vector of the mobile terminal is 40KB (that is, the output vector is 40960 int8, the dimension is [160, 16, 16]), the downstream task effect loss is within 1%, that is, the effect restoration rate is above 99%; (2) If the output vector of the mobile terminal is 20KB (that is, the output vector is 20480 int8, the dimension is [80, 16, 16]), the downstream task effect loss is within 3%, that is, the effect restoration rate is above 97%.

[0174] In summary, according to the technical solutions of the embodiments of this specification, on the one hand, the shallow network part of the large model is deployed on the mobile terminal, while the deep network part is deployed on the cloud, thereby building an end-cloud collaborative large model, which can solve the privacy leakage problem of the original data transmission. On the other hand, feature compression methods such as dimensionality reduction and quantization are used to perform feature compression on the output results of the shallow network on the mobile terminal. For example, the original 1280KB of data is converted into 20KB / 40KB of data, which reduces the amount of transmitted data; on the other hand, after the target content representation that is similar to the original content representation is obtained by dimensionality increase and restoration on the cloud, the effect loss of the downstream task can be controlled within a small range, for example, within 2%, and the effect restoration rate is high, for example, the effect restoration rate is greater than or equal to 98%.

[0175] On the other hand, this specification provides a non-transitory storage medium storing executable instructions. When the executable instructions are executed by a processor, the executable instructions instruct the processor to implement the steps of the end-cloud collaboration method for deep learning models described in this specification. In some possible implementations, various aspects of this specification can also be implemented in the form of a program product, which includes program code. When the program product is run on the electronic device 200, the program code is used to enable the electronic device 200 to perform the steps of the end-cloud collaboration method for deep learning models described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the electronic device 200. However, the program product of this specification is not limited to this. In this specification, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media include: an electrical connection having one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the electronic device 200, partially on the electronic device 200, as a stand-alone software package, partially on the electronic device 200 and partially on a remote computing device, or entirely on the remote computing device.

[0176] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented by way of example only and may not be limiting. Although not expressly stated herein, those skilled in the art will understand that this specification encompasses various reasonable changes, improvements, and modifications to the embodiments. Such changes, improvements, and modifications are intended to be suggested by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0178] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, “one embodiment,” “an embodiment,” and / or “some embodiments” mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is emphasized and should be understood that two or more references to “an embodiment,” “one embodiment,” or “an alternative embodiment” in various parts of this specification do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be appropriately combined in one or more embodiments of this specification.

[0179] It should be understood that in the foregoing descriptions of the embodiments of this specification, to facilitate understanding of a feature and to simplify this specification, various features are combined in a single embodiment, figure, or description thereof. However, this does not necessarily mean that these features are combined. When reading this specification, a person skilled in the art may label some of the devices as separate embodiments. In other words, the embodiments of this specification can also be understood as the integration of multiple sub-embodiments. This also applies when each sub-embodiment contains fewer than all the features of a single previously disclosed embodiment.

[0180] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, articles, and the like, cited herein is hereby incorporated by reference in its entirety for all purposes, except for any prosecution document history related thereto, any equivalent that may be inconsistent or conflicting with this document, or any equivalent prosecution document history that may have a limiting effect on the broadest scope of the claims now or hereafter associated with this document. For example, if there is any inconsistency or conflict between the descriptions, definitions, and / or usage of terms associated with any incorporated material and the terminology, descriptions, definitions, and / or usage associated with this document, the terminology in this document shall control.

[0181] Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can adopt alternative configurations based on the embodiments in this specification to implement the application in this specification. Therefore, the embodiments of this specification are not limited to the embodiments precisely described in the application.

Claims

1. A method for end-cloud collaboration of a deep learning model, wherein the deep learning model includes a shallow network deployed on the end-user side and a non-shallow network deployed on the cloud-user side, the method comprising: Extracting features of the content to be processed on the terminal side through the shallow network on the terminal side to obtain an original content representation corresponding to the content to be processed; Performing feature compression on the original content representation to obtain a compressed content representation corresponding to the original content representation; as well as The compressed content representation is sent to the cloud side, so that the compressed content representation is restored to the target content representation by the cloud side.

2. The method according to claim 1, wherein The terminal side is deployed with an encoding network, the encoding network is an independent network or a part of the shallow network, the feature compression includes feature dimensionality reduction, and the feature compression of the original content representation to obtain a compressed content representation corresponding to the original content representation includes: The encoding network is used to perform feature dimensionality reduction on the original content representation to obtain a reduced-dimensional content representation corresponding to the original content representation.

3. The method according to claim 2, wherein: The feature compression further includes quantization processing, wherein the feature compression of the original content representation to obtain a compressed content representation corresponding to the original content representation further includes: The quantization process is performed on each value of the feature vector of the reduced-dimensional content representation to obtain the compressed content representation corresponding to the original content representation.

4. The method according to claim 3, wherein: The reduced-dimensionality content representation is a floating-point feature vector, the compressed content representation is an integer feature vector, and the quantization processing is performed on each value of the reduced-dimensionality content representation to obtain the compressed content representation corresponding to the original content representation, including: The values ​​of the floating-point feature vector are quantized to obtain the integer feature vector corresponding to the original content representation.

5. The method according to claim 3, wherein The cloud side is deployed with a decoding network, which is an independent network or a part of the non-shallow network. The cloud side is used to restore the compressed content representation to the target content representation, including: The compressed content representation is feature decompressed by the decoding network on the cloud side to obtain the target content representation.

6. The method according to claim 5, wherein: The feature decompression includes quantization restoration processing, wherein the feature decompression of the compressed content representation is performed by the decoding network on the cloud side to obtain the target content representation, including: The quantized restoration process is performed on the compressed content representation through the decoding network on the cloud side to obtain the quantized restored content representation.

7. The method according to claim 6, wherein: The decoding network includes a dimension-raising subnetwork, the feature decompression further includes feature dimension-raising processing, and the feature decompression of the compressed content representation by the decoding network on the cloud side to obtain the target content representation further includes: The feature dimensionality increase processing is performed on the quantized restored content representation through the dimensionality increase sub-network to obtain the target content representation.

8. A method for end-cloud collaboration of a deep learning model, wherein the deep learning model includes a shallow network deployed on the end-user side and a non-shallow network deployed on the cloud-user side, the method comprising: Receiving a compressed content representation of the content to be processed sent by the terminal side, wherein the compressed content representation is a compressed content representation obtained by extracting and compressing features of the content to be processed by the shallow network on the terminal side; The compressed content representation is restored to a target content representation by the cloud side.

9. A model training method for a deep learning model, wherein the deep learning model includes a shallow network deployed on the terminal side and a non-shallow network deployed on the cloud side, the method comprising: Extracting features of the sample content through the shallow network on the terminal side to obtain an original sample content representation corresponding to the sample content; Performing feature compression on the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation; Sending the compressed sample content representation to the cloud side, so as to restore the compressed sample content representation to the target sample content representation through the cloud side; Based on the difference between the target sample content representation and the original sample content representation, a model loss of the deep learning model is determined to adjust model parameters of the deep learning model based on the model loss.

10. The method according to claim 9, wherein: The terminal side is deployed with an encoding network, which is an independent network or a part of the shallow network. The feature compression includes feature dimensionality reduction. The feature compression of the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation includes: The feature dimensionality reduction is performed on the original sample content representation through the encoding network to obtain a reduced-dimensional content representation corresponding to the original sample content representation.

11. The method according to claim 10, wherein: The feature compression further includes a clipping process, wherein the feature compression of the original sample content representation is performed to obtain a compressed sample content representation corresponding to the original sample content representation, and further includes: The clipping process is performed on each value of the feature vector of the dimensionality-reduced content representation to obtain a clipped content representation corresponding to the original sample content representation.

12. The method according to claim 10, wherein: The feature compression further includes quantization processing, wherein the feature compression of the original sample content representation to obtain a compressed sample content representation corresponding to the original sample content representation further includes: Determining a quantization scale factor corresponding to the reduced-dimensionality content representation; Based on the quantization scale factor, each value of the feature vector of the dimensionality reduction content representation is quantized to obtain the compressed sample content representation.

13. The method according to claim 12, wherein: The cloud side is deployed with a decoding network, which is an independent network or a part of the non-shallow network. The cloud side is used to restore the compressed sample content representation to the target sample content representation, including: The compressed sample content representation is feature decompressed by the decoding network on the cloud side to obtain the target sample content representation.

14. The method according to claim 13, wherein: The feature decompression includes quantization restoration processing, wherein the feature decompression of the compressed sample content representation is performed by the decoding network on the cloud side to obtain the target sample content representation, including: Performing the quantization restoration process on the compressed sample content representation through the decoding network on the cloud side to obtain a first quantization restoration content representation; Each value of the feature vector of the quantized restored content representation is multiplied by the quantization scale factor to obtain a second quantized restored content representation.

15. The method according to claim 14, wherein The decoding network includes a dimension-raising subnetwork, the feature decompression further includes feature dimension-raising processing, and the feature decompression of the compressed sample content representation by the decoding network on the cloud side to obtain the target sample content representation further includes: The feature dimensionality increasing process is performed on the second quantized restored content representation through the dimensionality increasing sub-network to obtain the target sample content representation.

16. The method according to claim 9, wherein The model loss includes feature difference and feature distance.

17. The method according to claim 16, wherein The determining of the model loss of the deep learning model based on the difference between the target sample content representation and the original sample content representation includes: Determining the feature difference and the feature distance between the feature vector representing the target sample content and the feature vector representing the original sample content; A weighted operation is performed on the feature difference and the feature distance to obtain a model loss of the deep learning model.

18. The method according to claim 16, wherein The characteristic difference is an absolute value difference, and the characteristic distance is a cosine distance.

19. An electronic device comprising: At least one storage medium storing at least one instruction set for end-cloud collaborative processing; as well as at least one processor, in communication with the at least one storage medium; Wherein, when the electronic device is running, the at least one processor reads the at least one instruction set and executes the end-cloud collaboration method of the deep learning model described in any one of claims 1-8 according to the instructions of the at least one instruction set.

20. An electronic device comprising: At least one storage medium storing at least one instruction set for performing model training processing; as well as at least one processor, in communication with the at least one storage medium; Wherein, when the device is running, the at least one processor reads the at least one instruction set and executes the model training method according to any one of claims 9-18 according to the instructions of the at least one instruction set.

Images