Image processing method and device based on cloud edge collaboration, electronic equipment and medium

Abstract

The application provides a cloud-edge collaboration-based image processing method and device, electronic equipment and medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring training image data; performing model segmentation processing on a preset original image processing model based on the training image data to obtain a first sub-model and a second sub-model; extracting first model parameters of the first sub-model and second model parameters of the second sub-model; sending the first model parameters to each edge terminal in a preset area; performing model optimization on the original image processing model based on the second model parameters to obtain a target image processing model; acquiring a target image sent by at least one edge terminal; performing deep feature extraction on the target image based on the target image processing model to obtain a target image representation vector; and performing target detection based on the target image representation vector to obtain target recognition data, wherein the target recognition data at least contains one target object. The application can improve the image processing efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method and apparatus, electronic device and medium based on cloud-edge collaboration. Background Technology

[0002] Current image processing methods often involve capturing images of various areas using a camera and then uploading the images to the cloud for processing. However, the image transmission process often takes a considerable amount of time, leading to significant time delays in image processing and impacting processing efficiency. Therefore, improving image processing efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to propose an image processing method, apparatus, electronic device, and medium based on cloud-edge collaboration, which aims to improve image processing efficiency.

[0004] To achieve the above objectives, a first aspect of this application proposes a cloud-edge collaborative image processing method applied in the cloud, the method comprising:

[0005] Acquire training image data;

[0006] Based on the training image data, the preset original image processing model is segmented to obtain a first sub-model and a second sub-model.

[0007] Extract the first model parameters of the first sub-model and the second model parameters of the second sub-model;

[0008] The first model parameters are sent to each edge of the preset region; wherein, the first model parameters are used to update the model at the edge.

[0009] Based on the second model parameters, the original image processing model is optimized to obtain the target image processing model;

[0010] Acquire at least one target image sent by the edge terminal; wherein the target image is obtained by the edge terminal according to the first model parameters;

[0011] Based on the target image processing model, deep feature extraction is performed on the target image to obtain the target image representation vector;

[0012] Target detection is performed based on the target image representation vector to obtain target recognition data, which contains at least one target object.

[0013] In some embodiments, the model segmentation processing of the preset original image processing model based on the training image data obtains a first sub-model and a second sub-model, including:

[0014] The preset original image processing model is located based on the training image data to obtain a target location of separation;

[0015] The original image processing model is segmented based on the target location of separation to obtain the first sub-model and the second sub-model.

[0016] In some embodiments, the preset original image processing model is located based on the training image data to obtain a target location of separation, including:

[0017] The number of levels of the original image processing model is obtained;

[0018] According to the number of levels, a candidate separation location of a preset separation point is determined, and the preset separation point is inserted into the candidate separation location;

[0019] The processing time of each level of the original image processing model on the training image data is detected;

[0020] Based on the candidate separation location where the preset separation point is located and the processing time, the target location of separation is selected from the candidate separation location.

[0021] In some embodiments, the original image processing model is segmented based on the target location of separation to obtain the first sub-model and the second sub-model, including:

[0022] The original image processing model is segmented based on the target location of separation;

[0023] The levels of the original image processing model before the target location of separation are combined to obtain the first sub-model;

[0024] The levels of the original image processing model after the target location of separation are combined to obtain the second sub-model.

[0025] In some embodiments, the target detection based on the target image feature vector obtains target recognition data, including:

[0026] The target image feature vector is shape-identified based on a preset shape category label to obtain a target shape category;

[0027] The target image feature vector is type-identified based on a preset object category label to obtain a target object category;

[0028] According to the target shape category and the target object category, a target object in the target image is determined, and target recognition data is obtained according to the target object.

[0029] In some embodiments, before the first model parameter is sent to each edge end of the preset region, the method further comprises:

[0030] Obtaining position data of all the edge ends and acquisition time of the training image data;

[0031] According to the acquisition time and a preset time threshold, a plurality of the edge ends are grouped;

[0032] According to the position data of the edge ends belonging to the same group, region division is performed to obtain the preset region.

[0033] To achieve the above object, a second aspect of the embodiment of the present application proposes an image processing method based on cloud-edge collaboration, applied to an edge end, the method comprising:

[0034] Receiving a first model parameter sent by the cloud, the first model parameter being obtained according to the image processing method based on cloud-edge collaboration of the first aspect;

[0035] According to the first model parameter, a preset model of the edge end is updated to obtain a local image processing model;

[0036] Obtaining an initial image;

[0037] Based on the local image processing model, image preprocessing is performed on the initial image to obtain a target image;

[0038] The target image is sent to the cloud.

[0039] To achieve the above object, a third aspect of the embodiment of the present application proposes an image processing device based on cloud-edge collaboration, applied to the cloud, the device comprising:

[0040] An acquisition module, configured to acquire training image data;

[0041] A segmentation module, configured to perform model segmentation processing on a preset original image processing model based on the training image data to obtain a first sub-model and a second sub-model;

[0042] A parameter extraction module, configured to extract a first model parameter of the first sub-model and a second model parameter of the second sub-model;

[0043] The parameter sending module is configured to send the first model parameter to each edge end of a preset area, wherein the first model parameter is used for the edge end to perform model updating.

[0044] The model optimization module is configured to perform model optimization on the original image processing model based on the second model parameter, to obtain a target image processing model.

[0045] The target image acquisition module is configured to acquire at least one target image sent by the edge end, wherein the target image is obtained by the edge end based on the first model parameter.

[0046] The feature extraction module is configured to perform deep feature extraction on the target image based on the target image processing model, to obtain a target image representation vector.

[0047] The target detection module is configured to perform target detection based on the target image representation vector, to obtain target recognition data, wherein the target recognition data at least contains one target object.

[0048] To achieve the above object, a fourth aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect or the method of the second aspect when executing the computer program.

[0049] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect or the method of the second aspect.

[0050] The cloud edge collaboration based image processing method, cloud edge collaboration based image processing device, electronic equipment and storage medium provided by the application obtain training image data; perform model segmentation processing on a preset original image processing model based on the training image data to obtain a first sub-model and a second sub-model; extract first model parameters of the first sub-model and second model parameters of the second sub-model; and send the first model parameters to each edge terminal in a preset area; wherein the first model parameters are used for model updating by the edge terminal; and perform model optimization on the original image processing model based on the second model parameters to obtain a target image processing model. This way can divide the original image processing model into two sub-models, so that the first sub-model and the second sub-model can be deployed in different positions (i.e. the first sub-model is deployed in the edge terminal, and the second sub-model is deployed in the cloud), improving the flexibility of model deployment in the image processing process, and effectively saving the image processing time of the model. Further, at least one target image sent by an edge terminal is obtained; wherein the target image is obtained by the edge terminal according to the first model parameters; deep feature extraction is performed on the target image based on the target image processing model to obtain a target image representation vector; target detection is performed based on the target image representation vector to obtain target recognition data, and the target recognition data contains at least one target object, which can realize the image processing steps with simple calculation but time-consuming transmission in the edge terminal, reduce the transmission time consumption in the image processing process, and effectively improve the efficiency of image processing. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of the cloud edge collaboration based image processing method provided by the embodiment of the application;

[0052] Figure 2 is a flowchart of step S102 in Figure 1

[0053] Figure 3 is a flowchart of step S201 in Figure 2

[0054] Figure 4 is a flowchart of step S202 in Figure 2

[0055] Figure 5 is another flowchart of the cloud edge collaboration based image processing method provided by the embodiment of the application;

[0056] Figure 6 is a flowchart of step S108 in Figure 1

[0057] Figure 7 is another flowchart of the cloud edge collaboration based image processing method provided by the embodiment of the application; ​​​​

[0058] Figure 8 Fig. 1 is a structural schematic diagram of an image processing device based on cloud-edge collaboration provided by an embodiment of the present application;

[0059] Figure 9 Fig. 2 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0061] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0063] First, the meanings of several terms involved in the present application are analyzed:

[0064] Artificial intelligence (AI): is a new technical science that studies, develops and applies systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Natural language processing (NLP): NLP uses computers to process, understand and use human languages (such as Chinese, English, etc.), which is a branch of artificial intelligence and an interdisciplinary subject of computer science and linguistics, also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research and language computing-related linguistic research related to language processing.

[0066] Information extraction (Information Extraction, NER): A text processing technology that extracts specified types of entities, relationships, events, etc. from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of specific units such as sentences, paragraphs, and chapters, and text information is composed of specific units such as words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, names, and places from text data is text information extraction, of course, the information extracted by the text information extraction technology can be various types of information.

[0067] Object detection (Object Detection): The task of object detection is to find all objects of interest in an image, determine their class and location, and is one of the core problems in computer vision. The core problems of object detection include four categories, namely (1) classification problem: which class does the image in the picture (or a certain region) belong to. (2) positioning problem: the target may appear anywhere in the image. (3) size problem: the target has various sizes. (4) shape problem: the target may have various shapes. Object detection is divided into two series: RCNN series and YOLO series, RCNN series is a representative algorithm based on region detection, and YOLO is a representative algorithm based on region extraction.

[0068] The current image processing method is often through the camera to take images of each region, and then upload the captured image to the cloud for image processing. The image transmission process often consumes a lot of time, which will cause a large time delay in image processing, affecting the image processing efficiency. Therefore, how to improve the image processing efficiency has become a technical problem to be solved.

[0069] Based on this, the embodiment of the application provides a cloud edge collaboration based image processing method, a cloud edge collaboration based image processing device, an electronic device and a storage medium, aiming to improve the efficiency of image processing.

[0070] The cloud edge collaboration based image processing method, the cloud edge collaboration based image processing device, the electronic device and the storage medium provided by the embodiment of the application are specifically described through the following embodiments. First, the cloud edge collaboration based image processing method in the embodiment of the application is described.

[0071] The embodiment of the application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain the best results.

[0072] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0073] The cloud edge collaboration based image processing method provided by the embodiment of the application relates to the field of artificial intelligence technology. The cloud edge collaboration based image processing method provided by the embodiment of the application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms; and the software can be an application for implementing the cloud edge collaboration based image processing method, etc., but is not limited to the above forms.

[0074] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0075] Figure 1 is an optional flowchart of the image processing method based on cloud edge collaboration provided by the embodiment of the application, applied to the cloud, Figure 1 The method in the cloud can include but is not limited to steps S101-S108.

[0076] Step S101, acquiring training image data;

[0077] Step S102, performing model segmentation processing on the preset original image processing model based on the training image data to obtain a first sub-model and a second sub-model;

[0078] Step S103, extracting first model parameters of the first sub-model and second model parameters of the second sub-model;

[0079] Step S104, sending the first model parameters to each edge end of a preset area; wherein the first model parameters are used for the edge end to perform model updating;

[0080] Step S105, performing model optimization on the original image processing model based on the second model parameters to obtain a target image processing model;

[0081] Step S106, acquiring at least one target image sent by an edge end; wherein the target image is obtained by the edge end according to the first model parameters;

[0082] Step S107, performing deep feature extraction on the target image based on the target image processing model to obtain a target image representation vector;

[0083] Step S108, performing target detection based on the target image representation vector to obtain target recognition data, the target recognition data containing at least one target object.

[0084] The steps S101 to S108 shown in the embodiments of the present application are as follows: training image data is acquired; a preset original image processing model is subjected to model segmentation processing based on the training image data, to obtain a first sub-model and a second sub-model; first model parameters of the first sub-model and second model parameters of the second sub-model are extracted; the first model parameters are sent to each edge end of a preset area; the first model parameters are used for model updating by the edge end; the original image processing model is subjected to model optimization based on the second model parameters, to obtain a target image processing model. This way can divide the original image processing model into two sub-models, so that the first sub-model and the second sub-model can be respectively deployed at different positions (i.e., the first sub-model is deployed at the edge end, and the second sub-model is deployed at the cloud end), the flexibility of model deployment in the image processing process is improved, and the image processing time of the model can be effectively saved. Further, at least one target image sent by the edge end is acquired; the target image is obtained by the edge end according to the first model parameters; deep feature extraction is performed on the target image based on the target image processing model, to obtain a target image feature vector; target detection is performed based on the target image feature vector, to obtain target recognition data, the target recognition data at least contains one target object, the image processing step with simple calculation but time-consuming transmission can be performed at the edge end, the transmission time consumption in the image processing process is reduced, and thus the efficiency of image processing is effectively improved.

[0085] It should be noted that the cloud-edge collaboration in the embodiments of the present application means that the edge end and the cloud end work cooperatively, i.e., the synergy of edge computing and cloud computing is realized, and data value is released together. The traditional cloud-edge collaboration mode is mainly as follows: after a terminal device generates data or a task request, the data is uploaded to an edge server through an edge network, and the edge server located in an edge computing center performs a computing task. A computing task with large amount of calculation and high complexity is migrated from the edge computing center to a cloud computing center through a core network, the cloud computing center completes big data analysis, and then stores the results and data to the cloud computing center or transmits the computing results, optimized output business rules and models to the edge computing center through the core network, the edge computing center transmits the computing results to the terminal device through the edge network, the edge computing performs business execution and optimization processing according to the new business rules transmitted by the cloud computing, and thus the cloud-edge collaboration is realized. The present application mainly trains a model in the cloud end, and transmits part of model parameters of a high-precision image processing model to the edge end, so that a preset model of the edge end can be updated based on the transmitted model parameters, the local image processing model obtained by the updating is used for edge computing, i.e., an initial image collected is subjected to image preprocessing, and a target image obtained by the preprocessing is uploaded to the cloud end, the resources of the cloud end are further used for cloud computing, and thus objects in the initial image are detected.

[0086] In step S101 of some embodiments, the cloud can obtain training image data from the edge or directly extract the training image data from a preset image database by issuing a data acquisition instruction, wherein the edge can be a mobile device such as a camera or a camera deployed at different locations, the training image data is an image captured by multiple edges within a preset time interval, and after receiving the data acquisition instruction issued by the cloud, the captured image is uploaded to the cloud; or the edge will capture an image every preset time interval and automatically upload the captured image to the cloud, and the cloud stores the image to the image database, and when image processing is needed, the cloud directly extracts the corresponding image data from the image database as the training image data.

[0087] Please refer to Figure 2 In some embodiments, step S102 can include but is not limited to steps S201 to S202:

[0088] Step S201, based on the training image data, the preset original image processing model is positioned and located to obtain a target separation position;

[0089] Step S202, based on the target separation position, the original image processing model is segmented and processed to obtain a first sub-model and a second sub-model.

[0090] In step S201 of some embodiments, the preset original image processing model can be a convolutional neural network model, etc., without limitation. When the original image processing model is positioned and located based on the training image data, the number of levels of the original image processing model needs to be counted first, and the candidate separation position for setting the separation point is determined according to the number of levels, wherein the candidate separation position should be selected between two levels, for example, the original image processing model includes four levels, i.e. convolution layer, activation layer, pooling layer and full connection layer, and there are three candidate separation positions, i.e. between the convolution layer and the activation layer, between the activation layer and the pooling layer, and between the pooling layer and the full connection layer. The original image processing model is divided into two sub-models at different candidate separation positions, and the image processing of the training image data is realized by using the two sub-models, and the processing time of the two sub-models is calculated respectively, the size of the processing time of the two sub-models at different candidate separation positions is compared, and the candidate separation position that makes the sum of the processing time of the two sub-models minimum is taken as the target separation position.

[0091] In step S202 of some embodiments, after the target separation position is determined, the original image processing model is divided into two sub-models, i.e. a first sub-model and a second sub-model, at the target separation position, so that the first sub-model and the second sub-model can be deployed at different positions, improving the flexibility of model deployment in the image processing process.

[0092] By the steps S201 to S202, the target separation position most suitable for separating the original image processing model can be conveniently identified, and the original image processing model is divided into two sub-models at the target separation position, so that the first sub-model and the second sub-model can be respectively deployed at different positions, the flexibility of model deployment in the image processing process is improved, and the image processing time of the model can be effectively saved.

[0093] Referring to Figure 3 In some embodiments, the step S201 can include but is not limited to steps S301 to S304.

[0094] In step S301, the number of levels of the original image processing model is obtained.

[0095] In step S302, the candidate separation positions of the preset separation points are determined according to the number of levels, and the preset separation points are inserted into the candidate separation positions.

[0096] In step S303, the processing time of each level of the original image processing model on the training image data is detected.

[0097] In step S304, the target separation position is selected from the candidate separation positions based on the candidate separation positions of the preset separation points and the processing time.

[0098] In step S301 of some embodiments, the number of levels of the original image processing model can be directly obtained according to the model structure and level type of the original image processing model, for example, the original image processing model includes a first convolution layer, a first activation layer, a first pooling layer, a first full connection layer, a second convolution layer, a second activation layer, a second pooling layer, a second full connection layer, a third convolution layer, a third activation layer, a fourth convolution layer and a fourth activation layer, and the number of levels of the original image processing model is 12.

[0099] In step S302 of some embodiments, when the candidate separation positions of the preset separation points are determined according to the number of levels, in order not to change the content structure of each level, the candidate separation positions should be selected between two levels, that is, the number of candidate separation positions should be less than the number of levels by 1, that is, when the number of levels of the original image processing model is 12, there are 11 candidate separation positions.

[0100] In step S303 of some embodiments, the candidate separation positions of the preset separation point can be taken as parameters, the preset separation point can be split into different candidate separation positions, the original image processing model can be split into two sub-models according to the different candidate separation positions, the two sub-models obtained by splitting can be used to process the training image data, and the inference delay (i.e., the image processing time of each level) of each level in the two sub-models to the input data can be calculated respectively. The image processing time of all levels of the two sub-models is summed to obtain the image processing total time corresponding to each candidate separation position.

[0101] In step S304 of some embodiments, the candidate separation positions of the preset separation point can be taken as parameters, and the image processing total time corresponding to each candidate separation position can be taken as an evaluation result. A regression model is constructed to obtain the relationship between the candidate separation positions of the preset separation point and the image processing total time. A regression curve is constructed according to the relationship between the candidate separation positions of the preset separation point and the image processing total time, so that the candidate separation position corresponding to the optimal image processing total time can be obtained more clearly, and this candidate separation position can be taken as the target segmentation position.

[0102] Through the above steps S301 to S304, the candidate separation positions that can be subjected to model segmentation can be determined according to the level number of the original image processing model, and the optimal candidate separation position can be screened out as the target separation position based on the image processing time of the model to the training image data, so that the accuracy of model segmentation can be effectively improved.

[0103] Please refer to Figure 4 In some embodiments, step S202 can include but is not limited to steps S401 to S403:

[0104] In step S401, the original image processing model is subjected to segmentation processing based on the target separation position.

[0105] In step S402, the levels before the target separation position in the original image processing model are subjected to level combination to obtain a first sub-model.

[0106] In step S403, the levels after the target separation position in the original image processing model are subjected to level combination to obtain a second sub-model.

[0107] In step S401 of some embodiments, the original image processing model is segmented based on the target segmentation position, and the original image processing model is divided into two parts. The activation value matrix of each level and the image processing time are plotted into a curve graph. According to the activation value matrix of each level and the image processing time, it can be known that the image processing time of the level before the target segmentation position is usually small, that is, the requirement of this part of the level for the computing resource is not high, but the activation value matrix of this part of the level is usually large, and the transmission time is long. The image processing time of the level after the target segmentation position is usually large, that is, the requirement of this part of the level for the computing resource is high, and the activation value matrix of this part of the level is usually small, and the transmission is convenient and the transmission time is short.

[0108] In step S402 of some embodiments, based on the curve graph of the activation value matrix of each level and the image processing time, since the image processing time of the level before the target segmentation position is small, the requirement of this part of the level for the computing resource is not high, but the activation value matrix of this part of the level is usually large, and the transmission time is long. Therefore, the levels before the target segmentation position in the original image processing model are combined to obtain a first sub-model, and the first sub-model is used for sending to the edge end for model configuration and used for processing the initial image obtained by the edge end, so as to reduce the influence of the transmission time on the overall image processing time.

[0109] In step S403 of some embodiments, based on the curve graph of the activation value matrix of each level and the image processing time, the image processing time of the level before the target segmentation position is usually large, that is, the requirement of this part of the level for the computing resource is high. Therefore, the levels after the target segmentation position in the original image processing model are combined to obtain a second sub-model, and the second sub-model is used for deploying the cloud end for image processing, which can effectively utilize the computing resource of the cloud end for image processing.

[0110] Through the above steps S401 to S403, the original image processing model can be conveniently segmented into a first sub-model and a second sub-model, and the two different sub-models can be respectively deployed to the edge end or the cloud end, so as to realize the deployment flexibility of the model, effectively shorten the data transmission time during model training and image processing, effectively reduce the image processing time when meeting the resource demand of image processing, and improve the efficiency of image processing.

[0111] In step S103 of some embodiments, the first model parameter of the first sub-model and the second model parameter of the second sub-model can be directly extracted by parameter derivation and the like. The first model parameter includes the model structure parameter, the learning rate, etc. of the first sub-model, and the second model parameter includes the model structure parameter, the learning rate, etc. of the second sub-model.

[0112] Referring to Figure 5 In some embodiments, before step S104, the cloud-edge collaborative image processing method can include, but is not limited to, steps S501-S503:

[0113] In step S501, position data of all edge ends and collection time of training image data are acquired.

[0114] In step S502, the plurality of edge ends are grouped according to the collection time and a preset time threshold.

[0115] In step S503, a preset region is obtained by region division according to the position data of the edge ends belonging to the same group.

[0116] In step S501 of some embodiments, the cloud end can acquire the position data of all edge ends from a preset database or by real-time collection, and acquire the collection time of the training image data according to a timer.

[0117] In step S502 of some embodiments, the time difference between the collection time of the training image data uploaded by each edge end is compared, and the edge ends with a time difference less than a preset time threshold are divided into the same group to obtain a plurality of edge end groups.

[0118] In step S503 of some embodiments, the grid region represented by the position data of the edge ends belonging to the same group is taken as the preset region by region division according to the position data of the edge ends belonging to the same group.

[0119] By the above steps S501-S503, the edge ends are regionally divided, the edge ends of the preset region are taken as one group, the plurality of edge ends are divided into a plurality of edge end groups, for an edge end group, the first model parameter generated by using the same model segmentation is used for model updating, all edge ends in the preset region can be updated according to the spatiotemporal locality of the edge end, the edge ends in the adjacent position can be configured by once cloud end model segmentation, and the resource consumption and time consumption of model segmentation training and model updating can be reduced.

[0120] In step S104 of some embodiments, the cloud can send the first model parameters to each edge terminal in the preset area through wired communication or wireless communication, so that all edge terminals in the preset area can perform model updating on the preset model according to the first model parameters to obtain a local image processing model, wherein the model structure and model performance of the local image processing model are basically the same as those of the first sub-model. This way enables the edge terminal to directly perform image preprocessing on the initial image after collecting the initial image containing the object, and performs the image processing step with simple calculation but time-consuming transmission on the edge terminal, which can reduce the transmission time consumption in the image processing process and improve the efficiency of image processing. At the same time, the embodiments of the present application can perform model updating on all edge terminals in the preset area according to the spatiotemporal locality of the edge terminal, so that the edge terminals in adjacent positions can be configured through one cloud model segmentation, which can reduce the resource consumption and time consumption of model segmentation training and model updating.

[0121] In step S105 of some embodiments, the original image processing model is optimized based on the second model parameters to simplify the model structure of the original image processing model to obtain a target image processing model, wherein the model structure and model performance of the target image processing model are basically the same as those of the second sub-model.

[0122] In step S106 of some embodiments, the cloud can obtain at least one target image sent by an edge terminal through wireless communication or the like; wherein the target image is obtained by the edge terminal according to the first model parameters, the target image is an image generated by the edge terminal performing image preprocessing on the collected initial image, and the target image contains object feature information photographed by the edge terminal.

[0123] In step S107 of some embodiments, the target image is subjected to deep feature extraction based on the target image processing model to realize convolution and pooling operations and the like on the target image, extract image information capable of representing object features in the target image, and obtain a target image representation vector, wherein the object features include object shape features, object category features, and the like.

[0124] Please refer to Figure 6 In some embodiments, step S108 includes but is not limited to steps S601 to S603:

[0125] Step S601, shape recognition is performed on the target image representation vector based on a preset shape category label to obtain a target shape category;

[0126] Step S602, type recognition is performed on the target image representation vector based on a preset object category label to obtain a target object category;

[0127] In step S603, the target object in the target image is determined according to the target shape category and the target object category, and target recognition data is obtained according to the target object.

[0128] In step S601 of some embodiments, the softmax function can be used to calculate the probability distribution of the target image feature vector on the preset multiple shape category labels, to obtain a first probability value corresponding to each shape category label. Since the size of the first probability value can clearly reflect the correlation between the target image feature vector and each shape category label, that is, the greater the first probability value corresponding to the shape category label, the higher the possibility that the target image feature vector belongs to this shape category label. Therefore, according to the size of the first probability value, the shape category label with the largest first probability value is selected as the target shape category of the initial image, wherein the shape category label includes a circle, a triangle, a diamond, a trapezoid, a cone, and other geometric shapes.

[0129] In step S602 of some embodiments, the softmax function can be used to calculate the probability distribution of the target image feature vector on the preset multiple object category labels, to obtain a second probability value corresponding to each object category label. Since the size of the second probability value can clearly reflect the correlation between the target image feature vector and each object category label, that is, the greater the second probability value corresponding to the object category label, the higher the possibility that the target image feature vector belongs to this object category label. Therefore, according to the size of the second probability value, the object category label with the largest second probability value is selected as the target object category of the initial image, wherein the object category label includes multiple levels of category labels, for example, the first level of category labels in the object category label includes people, animals, plants, vehicles, etc., the second level of category labels in the animal category includes cats, dogs, fish, and the vehicle category includes cars, bicycles, etc., without limitation.

[0130] In step S603 of some embodiments, the pixel region information where the target image feature vector is located is extracted from the target image, so that the target object can be displayed in the pixel region where the target image feature vector is located according to the object category label and the shape category label, the target detection of the target image is realized, and the target recognition data is obtained.

[0131] Through the above steps S601 to S603, the type and shape of the object in the target image can be conveniently identified according to the target image feature vector, so as to determine the target object existing in the target image, and the image detection accuracy can be effectively improved.

[0132] The image processing method based on cloud edge collaboration provided in the embodiments of the present application can obtain training image data, perform model segmentation processing on a preset original image processing model based on the training image data to obtain a first sub-model and a second sub-model, extract first model parameters of the first sub-model and second model parameters of the second sub-model, and send the first model parameters to each edge terminal in a preset area, wherein the first model parameters are used for model updating of the edge terminal. The original image processing model is optimized based on the second model parameters to obtain a target image processing model. This way can divide the original image processing model into two sub-models, so that the first sub-model and the second sub-model can be deployed at different positions (i.e., the first sub-model is deployed at the edge terminal, and the second sub-model is deployed at the cloud end), thereby improving the flexibility of model deployment in the image processing process and effectively saving the image processing time of the model. Further, at least one target image sent by an edge terminal is obtained, wherein the target image is obtained by the edge terminal according to the first model parameters. Deep feature extraction is performed on the target image based on the target image processing model to obtain a target image representation vector. Target detection is performed based on the target image representation vector to obtain target recognition data, and the target recognition data at least contains one target object, which can realize the image processing steps of simple calculation but time-consuming transmission at the edge terminal, reduce the transmission time consumption in the image processing process, and effectively improve the efficiency of image processing. Meanwhile, the embodiments of the present application can perform model updating on all edge terminals in the preset area according to the spatiotemporal locality of the edge terminal, so that the edge terminals in adjacent positions can be configured by one cloud end model segmentation, thereby reducing the resource consumption and time consumption of model segmentation training and model updating.

[0133] Figure 7 is an optional flowchart of the image processing method based on cloud edge collaboration provided in the embodiments of the present application, which is applied to an edge terminal, Figure 7 The method in the above can include but is not limited to steps S701 to S705.

[0134] Step S701, receiving first model parameters sent by a cloud end, wherein the first model parameters are obtained according to the image processing method based on cloud edge collaboration of the first aspect;

[0135] Step S702, updating a preset model of the edge terminal according to the first model parameters to obtain a local image processing model;

[0136] Step S703, obtaining an initial image;

[0137] Step S704, performing image preprocessing on the initial image based on the local image processing model to obtain a target image;

[0138] Step S705, sending the target image to the cloud end.

[0139] In step S701 of some embodiments, the edge end can receive the first model parameter sent by the cloud end through wired communication or wireless communication.

[0140] In step S702 of some embodiments, the edge end can perform model updating on the preset model according to the first model parameter, so that the preset model can have certain image processing functions, and obtain a local image processing model.

[0141] In step S703 of some embodiments, the edge end can perform information collection on the object appearing in the preset range based on real-time shooting, timing shooting, etc., and obtain an initial image containing object feature information.

[0142] In step S704 of some embodiments, after collecting the initial image containing the object, the edge end can directly perform image preprocessing on the initial image using the local image processing model, for example, based on the local image processing model to perform shallow image extraction, image standardization, etc., to improve the image quality, which is conducive to target detection by the cloud end on the image after image preprocessing, and can better improve the image processing effect.

[0143] In step S705 of some embodiments, the edge end can send the target image to the cloud end through wired communication or wireless communication, so that the cloud end can further process the target image, extract deep image features in the target image, and thus detect the target object in the initial image according to the image features.

[0144] The image processing method based on cloud-edge collaboration according to the embodiments of the present application receives the first model parameter sent by the cloud end through the edge end, updates the preset model of the edge end according to the first model parameter, obtains a local image processing model, performs image preprocessing on the collected initial image based on the local image processing model, obtains a target image, and finally uploads the target image to the cloud end, which can realize the image processing steps of simple calculation but time-consuming transmission at the edge end, reduce the transmission time consumption in the image processing process, and improve the efficiency of image processing.

[0145] Please refer to Figure 8 The embodiments of the present application also provide an image processing device based on cloud-edge collaboration, which is applied to the cloud end and can implement the above-mentioned image processing method based on cloud-edge collaboration. The device comprises:

[0146] The acquisition module 801 is configured to acquire training image data.

[0147] The segmentation module 802 is configured to perform model segmentation processing on the preset original image processing model based on the training image data, to obtain a first sub-model and a second sub-model.

[0148] The parameter extraction module 803 is configured to extract first model parameters of the first sub-model and second model parameters of the second sub-model.

[0149] The parameter sending module 804 is configured to send the first model parameters to each edge end of the preset area; wherein the first model parameters are used for model updating by the edge end.

[0150] The model optimization module 805 is configured to perform model optimization on the original image processing model based on the second model parameters, to obtain a target image processing model.

[0151] The target image acquisition module 806 is configured to acquire a target image sent by at least one edge end; wherein the target image is obtained by the edge end according to the first model parameters.

[0152] The feature extraction module 807 is configured to perform deep feature extraction on the target image based on the target image processing model, to obtain a target image representation vector.

[0153] The target detection module 808 is configured to perform target detection based on the target image representation vector, to obtain target recognition data, wherein the target recognition data contains at least one target object.

[0154] The specific implementation of the image processing device based on cloud-edge collaboration is basically the same as the specific embodiments of the above-mentioned image processing method based on cloud-edge collaboration, and will not be repeated here.

[0155] In addition, the embodiment of the present application also provides an image processing device based on cloud-edge collaboration, which is applied to an edge end and can realize the above-mentioned image processing method based on cloud-edge collaboration. The device comprises:

[0156] The parameter receiving module is configured to receive first model parameters sent by the cloud end, wherein the first model parameters are obtained according to the above-mentioned image processing device based on cloud-edge collaboration.

[0157] The preset model updating module is configured to update a preset model of the edge end according to the first model parameters, to obtain a local image processing model.

[0158] The initial image acquisition module is configured to acquire an initial image.

[0159] The image preprocessing module is configured to perform image preprocessing on the initial image based on the local image processing model, to obtain a target image.

[0160] The image sending module is configured to send the target image to the cloud end.

[0161] The specific implementation of the image processing device based on cloud-edge collaboration is basically the same as the specific embodiments of the above-mentioned image processing method based on cloud-edge collaboration, and will not be repeated here.

[0162] The embodiment of the present application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the cloud edge collaboration based image processing method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0163] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is shown, and the electronic device comprises:

[0164] The processor 901 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0165] The memory 902 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to realize the cloud edge collaboration based image processing method of the embodiments of the present application.

[0166] The input / output interface 903 is used to realize information input and output.

[0167] The communication interface 904 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0168] The bus 905 is used to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0169] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 realize the communication connection between each other in the device through the bus 905.

[0170] The embodiment of the present application further provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the cloud-edge collaborative image processing method.

[0171] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0172] The embodiment provided by the application provides a cloud edge collaboration-based image processing method, a cloud edge collaboration-based image processing device, an electronic device and a computer readable storage medium. The method comprises the following steps: obtaining training image data; performing model segmentation processing on a preset original image processing model based on the training image data, to obtain a first sub-model and a second sub-model; extracting first model parameters of the first sub-model and second model parameters of the second sub-model; and sending the first model parameters to each edge terminal in a preset area, wherein the first model parameters are used for model updating of the edge terminal. The method further comprises the following steps: performing model optimization on the original image processing model based on the second model parameters, to obtain a target image processing model. This way can divide the original image processing model into two sub-models, so that the first sub-model and the second sub-model can be deployed in different positions (i.e., the first sub-model is deployed in the edge terminal, and the second sub-model is deployed in the cloud), thereby improving the flexibility of model deployment in the image processing process and effectively saving the image processing time of the model. Further, the method comprises the following steps: obtaining at least one target image sent by an edge terminal, wherein the target image is obtained by the edge terminal based on the first model parameters; performing deep feature extraction on the target image based on the target image processing model, to obtain a target image representation vector; and performing target detection based on the target image representation vector, to obtain target recognition data, wherein the target recognition data comprises at least one target object. The method can realize the image processing steps with simple calculation and time-consuming transmission in the edge terminal, thereby reducing the transmission time consumption in the image processing process and effectively improving the efficiency of image processing. Meanwhile, the embodiment can perform model updating on all edge terminals in the preset area according to the spatiotemporal locality of the edge terminal, so that the edge terminals in adjacent positions can be configured with parameters through one cloud model segmentation, thereby reducing the resource consumption and time consumption of model segmentation training and model updating. The embodiment can reduce the time delay of the cloud and the edge terminal in the image processing as a whole, so that the high-precision image processing model can be trained in the cloud, and part of the structure of the image processing model can be deployed to the edge terminal, so that the edge terminal can process image data in real time, the cloud can perform more refined processing on the image data processed by the edge terminal, and the efficiency of image processing is improved.

[0173] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0174] Those skilled in the art can understand that, Figures 1-7 The technical solutions shown in the foregoing embodiments do not constitute a limitation on the embodiments of the application, and can comprise more or fewer steps, or combine certain steps, or different steps.

[0175] The apparatus embodiments described above are merely exemplary, and units described as separate components may or may not be physically separate, i.e., may be located in one place, or may be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0176] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0177] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so

[0178] It should be understood that in this application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time. Where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0179] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0180] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0181] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0182] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0183] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A cloud-edge collaboration based image processing method, characterized in that, Applied to the cloud, the method comprises: Obtaining training image data; Based on the training image data, the preset original image processing model is subjected to model segmentation processing to obtain a first sub-model and a second sub-model; Extracting the first model parameter of the first sub-model and the second model parameter of the second sub-model; the first model parameter is sent to each edge end of a preset area; wherein the first model parameter is used for model updating by the edge end; Based on the second model parameter, the original image processing model is subjected to model optimization to obtain a target image processing model; Obtaining at least one target image sent by the edge end; wherein the target image is obtained by the edge end according to the first model parameter; Based on the target image processing model, deep feature extraction is performed on the target image to obtain a target image feature vector; based on the target image feature vector, target detection is performed to obtain target recognition data, which at least contains one target object; The method further comprises: Obtaining the number of levels of the original image processing model; determining the candidate separation position of the preset separation point according to the number of levels, and inserting the preset separation point into the candidate separation position; detecting the processing time of each level of the original image processing model on the training image data; based on the candidate separation position where the preset separation point is located and the processing time, the target separation position is selected from the candidate separation position; Based on the target separation position, the original image processing model is subjected to segmentation processing to obtain the first sub-model and the second sub-model.

2. The image processing method of claim 1, wherein, The method further comprises: Based on the target separation position, the original image processing model is subjected to segmentation processing; The levels before the target separation position in the original image processing model are subjected to level combination to obtain the first sub-model; The levels after the target separation position in the original image processing model are subjected to level combination to obtain the second sub-model.

3. The image processing method of any of claims 1 to 2, wherein, The method further comprises: Based on the preset shape category label, shape recognition is performed on the target image feature vector to obtain a target shape category; Based on the preset object category label, type recognition is performed on the target image feature vector to obtain a target object category; According to the target shape category and the target object category, the target object in the target image is determined, and the target recognition data is obtained according to the target object.

4. The image processing method of any of claims 1 to 2, wherein, Before the first model parameter is sent to each edge end of a preset area, the method further comprises: Obtaining the position data of all edge ends and the collection time of the training image data; According to the collection time and the preset time threshold, a plurality of edge ends are grouped; According to the location data of the edge end belonging to the same group, the region division is performed to obtain the preset region.

5. An image processing method based on cloud-edge collaboration, characterized in that, The method is applied to the edge end, and the method comprises the following steps: receiving the first model parameter sent by the cloud end, wherein the first model parameter is obtained according to the cloud edge collaborative image processing method in any one of claims 1 to 4; updating a preset model of the edge end according to the first model parameter to obtain a local image processing model; obtaining an initial image; performing image preprocessing on the initial image based on the local image processing model to obtain a target image; sending the target image to the cloud end.

6. An image processing apparatus based on cloud-edge collaboration, characterized by, The device is applied to the cloud end, and the device comprises the following modules: an obtaining module, configured to obtain training image data; a segmentation module, configured to perform model segmentation processing on a preset original image processing model based on the training image data to obtain a first sub-model and a second sub-model; a parameter extraction module, configured to extract a first model parameter of the first sub-model and a second model parameter of the second sub-model; a parameter sending module, configured to send the first model parameter to each edge end in a preset region; wherein the first model parameter is used for model updating of the edge end; a model optimization module, configured to perform model optimization on the original image processing model based on the second model parameter to obtain a target image processing model; a target image obtaining module, configured to obtain a target image sent by at least one edge end; wherein the target image is obtained by the edge end according to the first model parameter; a feature extraction module, configured to perform deep feature extraction on the target image based on the target image processing model to obtain a target image representation vector; a target detection module, configured to perform target detection based on the target image representation vector to obtain target recognition data, wherein the target recognition data at least contains one target object; the model segmentation processing on the preset original image processing model based on the training image data to obtain the first sub-model and the second sub-model comprises the following steps: obtaining the number of levels of the original image processing model; determining the candidate separation position of the preset separation point according to the number of levels, and inserting the preset separation point into the candidate separation position; detecting the processing time of each level of the original image processing model on the training image data; based on the candidate separation position where the preset separation point is located and the processing time, screening out a target separation position from the candidate separation positions; performing segmentation processing on the original image processing model based on the target separation position to obtain the first sub-model and the second sub-model.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, and the memory stores a computer program, and the processor implements the following when executing the computer program: the cloud edge collaborative image processing method in any one of claims 1 to 4; or; the cloud edge collaborative image processing method in claim 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement: the cloud edge collaborative image processing method in any one of claims 1 to 4; or; the cloud edge collaborative image processing method in claim 5.

Images