Image processing method, device, equipment and medium based on end-edge-cloud collaboration

Abstract

The present disclosure provides an image processing method based on end-edge-cloud collaboration, which can be applied to the field of 5G technology. The method includes: in a cloud data center, according to a preset model cutting point, dividing a trained neural network model into a first part model and a second part model, and deploying the first part model and the second part model on a terminal device and an edge server respectively; in response to an image processing request sent by the terminal device to the cloud data center, the first part model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part model performs secondary processing on the intermediate result to generate an inference result; the cloud data center queries the real-time information corresponding to the inference result, and returns the real-time information to the terminal device. The present disclosure also provides an image processing device, electronic device, storage medium and program product based on end-edge-cloud collaboration.

Description

Technical Field

[0001] The present disclosure relates to the field of 5G technology, and more specifically to an image processing method, device, equipment and medium based on end-edge-cloud collaboration. Background Art

[0002] With the promotion of 5G technology and the "interconnection of everything" brought about by the Internet of Things technology, the rapid increase in network access devices has led to an explosive growth in the amount of data. Cloud computing faces challenges such as computing delays, congestion, low reliability, and security attacks. Edge computing builds a distributed development platform that integrates network, computing, storage, and application core capabilities at the edge of the network close to the object or data source, and realizes the sinking of computing tasks to the edge of the network, which can solve the shortcomings of cloud computing to a certain extent. At the same time, artificial intelligence has been recognized as an effective method for processing large-scale data, so the integration of artificial intelligence algorithms and edge computing has become a trend, which is called edge intelligence.

[0003] The booming development of mobile Internet has made it possible to provide users with a variety of services on portable devices. The application platform of technologies such as image processing has gradually expanded from PC to mobile. Sufficient computing resources, time resources and storage resources are the key to ensuring model performance. Therefore, the image processing model deployed on the mobile terminal must have the characteristics of small footprint, high accuracy and good real-time performance.

[0004] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technologies: At present, cloud computing architecture and edge computing architecture face many challenges in processing mobile terminal image services. The existing neural network model cannot meet both low resource consumption and high precision, while the computing power and storage space of the mobile terminal are often limited, and excessive time consumption will seriously reduce the customer experience. Therefore, directly deploying the existing model on the mobile terminal device with limited resources cannot meet the business needs. Summary of the invention

[0005] In view of this, the present disclosure provides an image processing method, apparatus, device and medium based on end-edge-cloud collaboration.

[0006] One aspect of the present disclosure provides an image processing method based on terminal-edge-cloud collaboration, including: in a cloud data center, dividing a trained neural network model into a first part model and a second part model according to a preset model cutting point, and deploying the first part model and the second part model on a terminal device and an edge server, respectively; in response to an image processing request sent by the terminal device to the cloud data center, the first part model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part model performs secondary processing on the intermediate result to generate an inference result; the cloud data center queries real-time information corresponding to the inference result, and returns the real-time information to the terminal device.

[0007] According to an embodiment of the present disclosure, the terminal device includes a mobile terminal device, and the mobile terminal device includes a mobile Internet of Things device.

[0008] According to an embodiment of the present disclosure, the preset model cutting point is comprehensively determined based on at least one preset performance indicator, wherein: the preset performance indicator includes the size of the first part of the model, the bandwidth consumption of data transmission between the terminal device and the edge server, the delay in completing an image processing request, or the image processing accuracy.

[0009] According to an embodiment of the present disclosure, the method also includes: the terminal device collects real-time information returned within a preset time range; and adjusts the preset model cutting point or the parameters of the neural network model according to the collected image processing request and the real-time information corresponding to the image processing request.

[0010] According to an embodiment of the present disclosure, the neural network model includes a lightweight neural network model, and the lightweight neural network model includes a Mobilenet-v1 network model, a Mobilenet-v2 network model, a Mobilenet-v3 network model, a ShuffleNet-v1 network model or a ShuffleNet-v2 network model.

[0011] According to an embodiment of the present disclosure, a neural network model is trained in a cloud data center in the following manner: pre-collecting image processing requests sent by terminal devices as sample data; inputting the sample data into the neural network model to obtain output data; re-inputting the output data into the neural network model for training iterations; repeating the step of re-inputting the output data into the neural network model for training iterations until the output data meets the preset image processing requirements, stopping the training iterations, and obtaining a trained neural network model.

[0012] According to an embodiment of the present disclosure, the first part of the model preliminarily processes the image processing request to obtain an intermediate result, including: the first part of the model extracts the image data in the image processing request, preprocesses the image data, and obtains the intermediate result; wherein the preprocessing includes at least one of rotation processing, scaling processing, cropping processing, grayscale processing, blur processing or compression coding.

[0013] According to an embodiment of the present disclosure, the terminal device is connected to the edge server via a wireless network, and the wireless network includes a 5G network.

[0014] According to an embodiment of the present disclosure, the image processing request includes a mobile terminal foreign exchange recognition request, a mobile terminal garbage classification request, or a digital recognition request.

[0015] Another aspect of the present disclosure provides an image processing device based on end-edge-cloud collaboration, including: a model cutting module, which is used to cut a trained neural network model into a first part model and a second part model according to a preset model cutting point in a cloud data center, and deploy the first part model and the second part model on a terminal device and an edge server, respectively; an end-edge processing module, which is used to respond to an image processing request sent by a terminal device to a cloud data center, the first part model preliminarily processes the image processing request to obtain an intermediate result, and the second part model performs secondary processing on the intermediate result to generate an inference result; a cloud processing module, which is used to query the cloud data center for real-time information corresponding to the inference result, and return the real-time information to the terminal device.

[0016] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned image processing method based on end-edge-cloud collaboration.

[0017] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the above-mentioned image processing method based on end-edge-cloud collaboration.

[0018] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and the instructions are used to implement the above-mentioned image processing method based on end-edge-cloud collaboration when executed.

[0019] Compared with the prior art, the image processing method, device, equipment and medium based on end-edge-cloud collaboration provided by the present disclosure have at least the following beneficial effects:

[0020] (1) Using a quantized neural network model not only reduces model resource requirements but also ensures image processing performance;

[0021] (2) When users use this model, the original data will be processed on the mobile terminal, protecting user privacy and preventing data leakage;

[0022] (3) The disclosure can achieve sustainable optimization. Users can choose to agree to provide feedback on input data and judgment results. At this time, the backend can collect production data, improve existing solutions, and achieve efficient deployment of models on mobile terminals under resource-constrained conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0024] Figure 1 The system architecture of conventional cloud computing and edge computing is schematically shown;

[0025] Figure 2 The system architecture of the image processing method and device based on end-edge-cloud collaboration according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 3 A flowchart of an image processing method based on end-edge-cloud collaboration according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 4 The following schematically shows an operation flow chart of collecting feedback on the inference results according to an embodiment of the present disclosure;

[0028] Figure 5 A flowchart of neural network model training according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 6 A block diagram schematically shows an image processing device based on end-edge-cloud collaboration according to an embodiment of the present disclosure; and

[0030] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0034] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some boxes or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or used in conjunction with an instruction execution system.

[0036] It should be noted that in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and do not violate public order and good morals.

[0037] Before describing the specific embodiments of the present disclosure in detail, the system architecture of cloud computing and edge computing is first explained to facilitate a better understanding of the present disclosure.

[0038] Figure 1 The system architecture of conventional cloud computing and edge computing is schematically shown.

[0039] Different types of Internet of Things (IoT) devices spread all over the world use various communication networks to build a complex network system that connects everything, serving many businesses such as health care and smart cities. At the same time, due to its large-scale data sets and high-density computing characteristics, deep learning has become the mainstream large-scale sensor data computing method. Deep learning models are generally deployed in cloud data centers with powerful computing and storage capabilities. Figure 1As shown in (a), in the conventional cloud computing architecture, data collected by massive devices are aggregated to the cloud computing data center for analysis and decision-making. However, the separate transmission of massive data will significantly increase the capacity pressure of the backhaul link and the computing load of the computing platform.

[0040] As an effective strategy to alleviate resource constraints and congestion, edge computing (Mobile Edge Computing, MEC) has become a new paradigm for solving localized computing needs in the Internet of Things. Edge computing realizes the sinking of data computing or data storage to the vicinity of end users. As a result, many computing nodes distributed on the network can keep data away from centralized data centers, reduce the computing pressure of cloud centers, and significantly reduce the waiting time for message exchange. In addition, compared with traditional cloud computing architecture, the distributed structure can balance network traffic and avoid traffic peaks in the network, reduce transmission delays between cloud servers and end users, and shorten the response time of real-time IoT applications. Figure 1 As shown in (b), the conventional edge computing architecture proposes to "sink" the task to the edge of the network for execution, which saves bandwidth resources and improves user experience. However, directly deploying the model on the edge node will be constrained by the limited computing and storage resources of the edge node, resulting in the inability to process large-scale data in real time, which reduces the user experience.

[0041] In view of this, the embodiments of the present disclosure provide an image processing method based on end-edge-cloud collaboration, which can be applied to the field of 5G technology. The method includes: in the cloud data center, according to the pre-model cutting point, the trained neural network model is divided into a first part model and a second part model, and the first part model and the second part model are deployed on the terminal device and the edge server respectively; in response to the image processing request sent by the terminal device to the cloud data center, the first part model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part model performs secondary processing on the intermediate result to generate an inference result; the cloud data center queries the real-time information corresponding to the inference result, and returns the real-time information to the terminal device.

[0042] Figure 2 The system architecture of the image processing method and device based on end-edge-cloud collaboration according to an embodiment of the present disclosure is schematically shown. It should be noted that Figure 2 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0043] like Figure 2As shown, the system architecture according to this embodiment may include a terminal device, an edge server, and a cloud data center. The terminal device and the edge server may communicate with each other via a communication link, and the edge server and the cloud data center may also communicate with each other via a communication link. The communication link may include various connection types, such as wired and / or wireless communication links, etc.

[0044] The terminal device may be any electronic device that supports web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. In particular, the terminal device may include a mobile terminal device, and more specifically, the mobile terminal device may include a mobile Internet of Things device. A mobile Internet of Things device is an Internet of Things terminal based on a mobile handheld device, such as an autonomous vehicle, a smart home appliance, a smart access control, a video terminal device, etc.

[0045] Users can use terminal devices to interact with edge servers through communication links to receive or send messages, etc. Similarly, edge servers can also interact with cloud data centers through communication links to receive or send messages, etc. Various communication client applications can be installed on terminal devices, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only for example).

[0046] The cloud data center can be a data center, a cloud computing center or a supercomputing center, which has powerful computing and storage capabilities and can provide various management or services, such as a backend management server that provides support for websites browsed by users using terminal devices (only as an example). The backend management server can analyze and process the received user request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0047] The present invention combines edge computing with cloud computing, respectively utilizing the advantages of low data transmission latency of edge computing and powerful computing power of cloud computing, and realizes the process of accelerating the use of image data collected by mobile devices to train deep learning models for image recognition through collaboration between mobile devices and edge servers and cloud data centers.

[0048] Figure 3 The flowchart of the image processing method based on end-edge-cloud collaboration according to an embodiment of the present disclosure is schematically shown.

[0049] like Figure 3 As shown, the image processing method based on end-edge-cloud collaboration may include operations S301 to S303.

[0050] In operation S301, the trained neural network model is divided into a first model part and a second model part according to a preset model cutting point in the cloud data center, and the first model part and the second model part are deployed on the terminal device and the edge server respectively.

[0051] In operation S302, in response to an image processing request sent by the terminal device to the cloud data center, the first part of the model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part of the model performs secondary processing on the intermediate result to generate an inference result.

[0052] In operation S303, the cloud data center queries the real-time information corresponding to the inference result and returns the real-time information to the terminal device.

[0053] It is understandable that for common neural network models, such as convolutional neural networks, it is difficult to directly train the model when the terminal device resources are limited. Figure 2 and Figure 3 , the present disclosure divides the entire neural network model according to performance indicators such as the computing power or network bandwidth of the terminal device, so that the terminal device only needs to calculate a certain number of continuous network layers starting from the first layer, and the edge server handles the calculation of the remaining continuous network layers in the entire neural network model except for the continuous network layers of the terminal device, and the cloud data center is only used to query the real-time information stored in the database. In addition, the embodiment of the present disclosure pre-trains the neural network model in the cloud data center to obtain the trained entire neural network model for subsequent model segmentation and deployment. Therefore, the embodiment of the present disclosure takes advantage of the significant advantages of end-edge-cloud collaborative computing.

[0054] Combination Figure 2 and Figure 3 ,The present disclosure makes full use of the available resources in the device-edge-cloud hierarchy, saving the computing, storage, and wireless resources occupied by the model while ensuring the ,quality of service, thereby achieving efficient deployment of the model on the mobile terminal under ,resource-constrained conditions.

[0055] In an embodiment of the present disclosure, the preset model cutting point is comprehensively determined based on at least one preset performance indicator, wherein: the preset performance indicator includes the size of the first part of the model, the bandwidth consumption of data transmission between the terminal device and the edge server, the delay in completing an image processing request, or the image processing accuracy.

[0056] Therefore, by comparing multiple cutting point segmentation schemes, the cutting point with the best overall performance is selected to obtain the preset model cutting point. This method of comprehensive selection of preset performance indicators cuts the model based on the image processing performance indicators of each node, which can effectively allow some neural network models to run on terminal devices or edge servers near terminal devices, better adapt to terminal devices and edge servers with different computing capabilities, and achieve the best end-edge-cloud collaborative image processing effect.

[0057] Figure 4 The following schematically shows an operation flow chart of collecting feedback on inference results according to an embodiment of the present disclosure.

[0058] like Figure 4 As shown, after operation S303, the image processing method based on end-edge-cloud collaboration of the embodiment of the present disclosure may further include the following operations S410 to S420.

[0059] In operation S410, the terminal device collects real-time information returned within a preset time range.

[0060] In operation S420, the preset model cutting points or the parameters of the neural network model are adjusted according to the collected image processing requests and the real-time information corresponding to the image processing requests.

[0061] Through the above embodiments, if the user of the terminal device agrees that the background collects usage information, the background of the terminal device can analyze the deficiencies of the existing neural network model based on the real-time data actually generated, further optimize the neural network model and the preset model cutting points, thereby optimizing the image processing performance of the cloud data center. Furthermore, the technical solution disclosed in this disclosure can achieve sustainable optimization.

[0062] In the disclosed embodiment, the neural network model includes a lightweight neural network model, and the lightweight neural network model includes a Mobilenet-v1 network model, a Mobilenet-v2 network model, a Mobilenet-v3 network model, a ShuffleNet-v1 network model or a ShuffleNet-v2 network model.

[0063] It is understandable that due to the limitation of hardware resources and computing power, it is difficult for mobile devices to run complex deep learning network models. MobileNet series network models or ShuffleNet series network models make it possible for mobile terminals or embedded devices to run neural network models. However, the embodiments of the present disclosure are not limited thereto, and other types of lightweight neural network models can be used as the neural network model of this embodiment as needed.

[0064] Through the above embodiments, the present disclosure uses a quantized neural network model, and the neural network model can be used on a portable device to reduce the amount of calculation and the amount of parameters of the model, thereby reducing the model resource requirements and ensuring image processing performance.

[0065] In the disclosed embodiment, the terminal device includes a mobile terminal device, and the mobile terminal device includes a mobile Internet of Things device. Therefore, when the user of the terminal device uses the neural network model for preliminary processing, the original data will be processed on the mobile terminal, protecting the user's privacy and preventing data leakage.

[0066] Figure 5 The flowchart of neural network model training according to an embodiment of the present disclosure is schematically shown.

[0067] like Figure 5 As shown, the neural network model is trained in the cloud data center through the following operations S510 to S540.

[0068] In operation S510, image processing requests sent by terminal devices are collected in advance as sample data.

[0069] In operation S520, the sample data is input into the neural network model to obtain output data.

[0070] In operation S530, the output data is re-input into the neural network model for training iterations.

[0071] In operation S540, the step of re-inputting the output data into the neural network model for training iteration is repeated until the output data meets the preset image processing requirements, and the training iteration is stopped to obtain a trained neural network model.

[0072] In an embodiment of the present disclosure, the first part of the model preliminarily processes the image processing request to obtain an intermediate result, including: the first part of the model extracts the image data in the image processing request, preprocesses the image data, and obtains the intermediate result; wherein the preprocessing includes at least one of rotation processing, scaling processing, cropping processing, grayscale processing, blur processing or compression coding.

[0073] Through the preliminary processing of the terminal device, the embodiment of the present disclosure takes into account the limited computing power of the terminal device, and can perform pre-processing of the input image data in the image processing request in accordance with the image processing capability of the terminal device before sending it to the edge server for secondary processing. The pre-processing may include rotating, scaling, cropping, grayscale processing, blurring, or compression encoding the input image data, so that the input image data is regularized into an intermediate result that meets the input requirements of the edge server.

[0074] In the embodiment of the present disclosure, the terminal device is connected to the edge server via a wireless network, and the wireless network includes a 5G network. Therefore, the present disclosure takes into account the huge wireless resource consumption caused by the massive data transmission under the 5G network and the high demand for computing and storage resources of the deep neural network, and realizes the efficient deployment of the neural network model on the mobile terminal under the condition of limited resources.

[0075] In an embodiment of the present disclosure, according to an embodiment of the present disclosure, the image processing request includes a mobile terminal foreign exchange recognition request, a mobile terminal garbage classification request, or a digital recognition request.

[0076] For example, under a mobile foreign exchange identification request, a mobile user uploads a picture of a foreign currency. After the collaborative processing of the first part model and the second part model, the edge server obtains the inference result and sends it to the cloud data center. The cloud data center will query the real-time exchange rate corresponding to the inference result and send it to the mobile user.

[0077] Through the above embodiments, the present invention adopts a device-edge-cloud collaborative reasoning method in the scenario where a mobile user sends an image processing request or a real-time search, and makes full use of the available resources in the terminal device-edge node-cloud data center hierarchy to optimize the overall performance of the network model and realize edge intelligent computing.

[0078] From the above, it can be seen that the embodiments of the present disclosure propose a new edge intelligent computing architecture, which further optimizes the neural network by utilizing cloud-based training of deep neural network models, collaborative reasoning of terminal devices, edge servers, and cloud-based data centers, and collecting feedback information in the cloud.

[0079] Through the above-mentioned image processing method based on device-edge-cloud collaboration, the present disclosure also provides an image processing device based on device-edge-cloud collaboration. Figure 6 The device is described in detail.

[0080] Figure 6 A block diagram of an image processing device based on end-edge-cloud collaboration according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 6 As shown, the image processing device 600 based on end-edge-cloud collaboration may include a model cutting module 610 , an end-edge processing module 620 , and a cloud processing module 630 .

[0082] A model cutting module 610 is used to cut the trained neural network model into a first model and a second model according to a preset model cutting point in a cloud data center, and deploy the first model and the second model on a terminal device and an edge server respectively;

[0083] The edge processing module 620 is used to respond to the image processing request sent by the terminal device to the cloud data center. The first part of the model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part of the model performs secondary processing on the intermediate result to generate an inference result;

[0084] The cloud processing module 630 is used for the cloud data center to query the real-time information corresponding to the inference result and return the real-time information to the terminal device.

[0085] It should be noted that the device part in the embodiments of the present disclosure corresponds to the method part in the embodiments of the present disclosure. The description of the image processing device part based on end-edge-cloud collaboration specifically refers to the image processing method part based on end-edge-cloud collaboration, which will not be repeated here.

[0086] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0087] For example, any of the model cutting module 610, the edge processing module 620, and the cloud processing module 630 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the model cutting module 610, the edge processing module 620, and the cloud processing module 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the model cutting module 610, the edge processing module 620 and the cloud processing module 630 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0088] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 7 As shown, the electronic device 700 includes a processor 710 and a computer-readable storage medium 720. The electronic device 700 can execute the image processing method based on device-edge-cloud collaboration according to an embodiment of the present disclosure.

[0090] Specifically, the processor 710 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 710 may also include an onboard memory for cache purposes. The processor 710 may be a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0091] The computer-readable storage medium 720 may be, for example, a non-volatile computer-readable storage medium, and specific examples include but are not limited to: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a memory, such as a random access memory (RAM) or flash memory; and the like.

[0092] The computer-readable storage medium 720 may include a computer program 721 , which may include code / computer-executable instructions that, when executed by the processor 710 , cause the processor 710 to perform a method according to an embodiment of the present disclosure or any variation thereof.

[0093] The computer program 721 may be configured to have, for example, a computer program code including a computer program module. For example, in an example embodiment, the code in the computer program 721 may include one or more program modules, for example, including 721A, module 721B, ... It should be noted that the division method and number of modules are not fixed, and those skilled in the art may use appropriate program modules or program module combinations according to actual conditions, and when these program module combinations are executed by the processor 710, the processor 710 may execute the method according to the embodiment of the present disclosure or any variation thereof.

[0094] According to an embodiment of the present disclosure, at least one of the model cutting module 610, the edge processing module 620, and the cloud processing module 630 can be implemented as a reference. Figure 7 The computer program modules described herein, when executed by the processor 710, can implement the corresponding operations described above.

[0095] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the image processing method based on end-edge-cloud collaboration according to the embodiment of the present disclosure is implemented.

[0096] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0097] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the image processing method based on end-edge-cloud collaboration provided by the embodiment of the present disclosure.

[0098] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0099] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0101] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing method based on device-edge-cloud collaboration, comprising: In the cloud data center, according to the preset model cutting point, the trained neural network model is divided into a first part model and a second part model, and the first part model and the second part model are deployed on the terminal device and the edge server respectively; In response to an image processing request sent by a terminal device to a cloud data center, the first part model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part model performs secondary processing on the intermediate result to generate an inference result; The cloud data center queries the real-time information corresponding to the inference result, and returns the real-time information to the terminal device; as well as The terminal device collects real-time information returned within a preset time range; According to the collected image processing requests and the real-time information corresponding to the image processing requests, the preset model cutting points or the parameters of the neural network model are adjusted.

2. The method according to claim 1, wherein: The terminal device includes a mobile terminal device, and the mobile terminal device includes a mobile Internet of Things device.

3. The method according to claim 1, wherein: The preset model cutting point is comprehensively determined according to at least one preset performance indicator, wherein: The preset performance indicators include the size of the first part model, the bandwidth consumption of data transmission between the terminal device and the edge server, the delay of completing an image processing request or the image processing accuracy.

4. The method according to claim 1, wherein: The neural network model includes a lightweight neural network model, and the lightweight neural network model includes a Mobilenet-v1 network model, a Mobilenet-v2 network model, a Mobilenet-v3 network model, a ShuffleNet-v1 network model or a ShuffleNet-v2 network model.

5. The method according to claim 1, wherein: The neural network model is trained in the cloud data center in the following manner: Preliminarily collecting image processing requests sent by the terminal device as sample data; Inputting the sample data into a neural network model to obtain output data; Re-inputting the output data into the neural network model for training iterations; Repeat the step of re-inputting the output data into the neural network model for training iteration until the output data meets the preset image processing requirements, stop the training iteration, and obtain a trained neural network model.

6. The method according to claim 1, wherein: The first part of the model preliminarily processes the image processing request to obtain an intermediate result, including: The first partial model extracts the image data in the image processing request, and obtains an intermediate result after preprocessing the image data; The preprocessing includes at least one of rotation processing, scaling processing, cropping processing, grayscale processing, blur processing or compression coding.

7. The method according to claim 1, wherein: The terminal device is connected to the edge server via a wireless network, and the wireless network includes a 5G network.

8. The method according to claim 1, wherein: The image processing request includes a mobile terminal foreign exchange recognition request, a mobile terminal garbage classification request or a digital recognition request.

9. An image processing device based on device-edge-cloud collaboration, comprising: A model cutting module is used to cut the trained neural network model into a first part model and a second part model according to a preset model cutting point in a cloud data center, and deploy the first part model and the second part model on a terminal device and an edge server respectively; The edge processing module is used to respond to the image processing request sent by the terminal device to the cloud data center, the first part model performs preliminary processing on the image processing request to obtain an intermediate result, and the second part model performs secondary processing on the intermediate result to generate an inference result; A cloud processing module, used for the cloud data center to query the real-time information corresponding to the inference result and return the real-time information to the terminal device; as well as The terminal device collects real-time information returned within a preset time range; According to the collected image processing requests and the real-time information corresponding to the image processing requests, the preset model cutting points or the parameters of the neural network model are adjusted.

10. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 8 when executed.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images