Image data processing method, device, apparatus and storage medium

Abstract

Embodiments of the application disclose an image data processing method, device and equipment and a storage medium. The method can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises the following steps: obtaining a target image recognition model used for recognizing a target image; the target image recognition model comprises M network processing layers; determining the abstract degree of the M network processing layers to image features in the target image; dividing the M network processing layers according to the corresponding abstract degrees of the M network processing layers to obtain a first image recognition sub-model and a second image recognition sub-model; deploying the first image recognition sub-model in a terminal used for cooperatively deploying the target image recognition model, and deploying the second image recognition sub-model in a server used for cooperatively deploying the target image recognition model. The application improves the accuracy of image recognition while avoiding leakage of private data in the image.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, vehicle networking and other technologies, and in particular to an image data processing method, apparatus, device and storage medium. Background Technology

[0002] Currently, in image recognition scenarios, to prevent the leakage of privacy data in the original image, the terminal usually needs to encrypt the original image, upload the encrypted original image to the server, and then the server processes the encrypted original image to obtain the recognition result. In practice, it has been found that because the content in the encrypted original image has a certain degree of distortion, directly processing the encrypted original image will lead to low accuracy of the recognition result. Summary of the Invention

[0003] This application provides an image data processing method, apparatus, device, and storage medium that improves the accuracy of image recognition while avoiding the leakage of private data in images.

[0004] One embodiment of this application provides an image data processing method, including:

[0005] Obtain a target image recognition model for recognizing the target image; the target image recognition model includes M network processing layers;

[0006] Determine the degree of abstraction of the image features in the target image by each of the above M network processing layers;

[0007] Based on the level of abstraction corresponding to the above M network processing layers, the above M network processing layers are divided to obtain the first image recognition sub-model and the second image recognition sub-model;

[0008] The first image recognition sub-model is deployed in a terminal for collaborative deployment of the target image recognition model, and the second image recognition sub-model is deployed in a server for collaborative deployment of the target image recognition model. The terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image, and the server is used to obtain the low-level feature map from the terminal, call the second image recognition sub-model to perform object recognition on the low-level feature map, and obtain the object attributes of the target object in the target image.

[0009] One embodiment of this application provides an image data processing apparatus, including:

[0010] The acquisition module is used to acquire the target image recognition model for recognizing the target image, and the image processing performance parameters of the terminal for collaboratively deploying the target image recognition model; the target image recognition model includes M network processing layers.

[0011] The determination module is used to determine the degree of abstraction of the image features in the target image by the above M network processing layers;

[0012] The partitioning module is used to partition the M network processing layers according to the abstraction level of the M network processing layers and the image processing performance parameters of the terminal, so as to obtain the first image recognition sub-model and the second image recognition sub-model.

[0013] The deployment module is used to deploy the first image recognition sub-model on the terminal and the second image recognition sub-model on the server used for collaborative deployment of the target image recognition model. The terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image. The server is used to obtain the low-level feature map from the terminal and call the second image recognition sub-model to perform object recognition on the low-level feature map to obtain the object attributes of the target object in the target image.

[0014] One embodiment of this application provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0015] One embodiment of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] One embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0017] In this application, the target image recognition model is adaptively divided based on the level of abstraction of image features in the target image across its various network processing layers. This allows for the deployment of a portion of the network processing layers (i.e., the first image recognition sub-model) on the terminal and another portion (i.e., the second image recognition sub-model) on the server, achieving collaborative deployment of the target image recognition model. This eliminates the need to deploy all network processing layers on the terminal, reducing the image processing load on the terminal. The level of abstraction here reflects the difficulty for a user to distinguish the target object (or object attributes) in the target image from the image features output by the network processing layer. A higher level of abstraction makes it more difficult for the user to distinguish the target object from the image features output by the network processing layer; conversely, a lower level of abstraction makes it easier for the user to distinguish the target object from the image features output by the network processing layer. The first image recognition sub-model, based on this level of abstraction, outputs a low-level feature map of the target image. The target object (or the object attribute of the target object) in the target image cannot be distinguished from this low-level feature map. Therefore, when the terminal needs to identify the object attribute of the target object in the target image, the terminal only needs to upload the low-level feature map to the server. The server then uses the second image recognition sub-model to perform object recognition on the low-level feature map and obtain the object attribute of the target object. The terminal does not need to upload the original target image to the server. This can avoid the leakage of privacy data in the target image and improve the accuracy of image recognition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the architecture of an image data processing system provided in this application;

[0020] Figure 2 This is a flowchart illustrating an image data processing method provided in this application;

[0021] Figure 3 This is a schematic diagram of the interaction scenario between various devices in an image data processing system provided in this application;

[0022] Figure 4 This is a schematic diagram of the interaction scenario between various devices in an image data processing system provided in this application;

[0023] Figure 5 This is a flowchart illustrating an image data processing method provided in this application;

[0024] Figure 6 This is a flowchart illustrating an image data processing method provided in this application;

[0025] Figure 7 This is a schematic diagram of a scenario for training an initial image recognition model provided in this application;

[0026] Figure 8 This is a schematic diagram of the structure of a target image recognition model provided in this application;

[0027] Figure 9 This is a schematic diagram of the structure of an image data processing device provided in an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] This application is mainly applied in various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. For example, computer equipment can use artificial intelligence technology to identify vehicles in a target image, obtain the number of vehicles in the road, determine the traffic flow based on the number of vehicles in the road, and then realize vehicle management based on the traffic flow.

[0031] Understandably, Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI essentially studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0032] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0033] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0034] To facilitate a clearer understanding of this application, the image data processing method for implementing this application is first introduced, such as... Figure 1 As shown, the image data processing includes a server 10 and a terminal cluster. The terminal cluster can include one or more terminals; the number of terminals is not limited here. Figure 1 As shown, the terminal cluster may specifically include terminal 1, terminal 2, ..., terminal n; it can be understood that terminal 1, terminal 2, terminal 3, ..., terminal n can all connect to server 10 via the network so that each terminal can interact with server 10 via the network.

[0035] Understandably, the target terminal and server can be devices used for collaborative deployment of the target image recognition model. The target terminal can handle a small portion of the computational load for deploying the target image recognition model, while the server handles the majority of the computational load. Here, the target terminal can be any terminal in the aforementioned terminal cluster. Specifically, the target terminal is used to deploy the first image recognition sub-model of the target image recognition model, and the server is used to deploy the second image recognition sub-model. The first image recognition sub-model extracts the low-level feature map of the image, and the second image recognition sub-model performs object recognition on the low-level feature map to obtain the object attributes of the target object in the target image. The server then returns the object attributes of the target object to the target terminal. In this way, after the terminal obtains the target image, it only needs to upload the low-level feature map of the target image to the server, without needing to upload the original target image, thus avoiding the leakage of private data in the target image. By collaboratively deploying the target image recognition model through the target terminal and server, the limitation of the target terminal's image processing performance—which can be limited to deploying only lightweight target image recognition models (i.e., non-lightweight models)—is avoided, allowing for the use of non-lightweight models to recognize the target image, thereby improving the accuracy of image recognition.

[0036] Understandably, as personal privacy data receives more and more attention and relevant laws and regulations are implemented, the protection of privacy data has more explicit requirements. For example, information (such as images) obtained by mobile phones, vehicles, robots and other collection terminals need to have privacy data erased before being handed over to downstream users. The privacy computing method proposed in this invention has, but is not limited to, the following product manifestations: (1) Mobile phone face verification and recognition. Many mobile phone applications have face recognition login functions. Face recognition requires the image recognition model to have high accuracy to avoid face misrecognition and ensure the security of login information. However, due to the limited computing power of mobile phones, it is difficult to directly deploy high-precision image recognition models with large computing power. Using the method of this application, a small amount of computing power can be used to extract the low-level feature map of the face on the mobile phone, and then the low-level feature map without the original face information is transmitted to the remote end (such as the server) for further reasoning to determine whether the face recognition is successful. Finally, the recognition result is fed back to the mobile phone side.

[0037] (2) Training Data Feedback. Autonomous driving and advanced driver assistance systems require image recognition models to detect targets such as pedestrians, vehicles, lane lines, traffic signs, traffic lights, and drivable areas in driving scenarios. The development and iteration of these image recognition models require a large amount of image data, which cannot be met by relying solely on dedicated image data collection vehicles. Therefore, images captured by mass-produced vehicles can be fed back. Due to the large number and wide distribution of mass-produced vehicles, the quantity and diversity of image data are well guaranteed. However, images fed back from the vehicle may contain privacy or sensitive information such as license plates and faces, requiring desensitization processing. Using the method proposed in this application, the low-level feature map of the image can be extracted at the vehicle with low computational cost. After the low-level feature map is transmitted to the cloud for further inference, the perception results are fed back to the vehicle. Based on the perception results, the vehicle decrypts the privacy or sensitive information areas within the image and then transmits the decrypted image to the cloud for subsequent annotation and training.

[0038] The server can be a single physical server, a server cluster or distributed system consisting of at least two physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal can specifically refer to in-vehicle terminals, smartphones, tablets, laptops, desktop computers, smart speakers, speakers with screens, smartwatches, etc., but is not limited to these. The terminals and servers can be directly or indirectly connected via wired or wireless communication. The number of terminals and servers can be one or at least two; this application does not impose any restrictions.

[0039] It is understandable that the image data processing system described above can implement the image data processing method in this application. Taking the application scenario of the Internet of Vehicles as an example, such as... Figure 2 , Figure 3 and Figure 4 The image processing method of this application will be described as follows: Figure 2 , Figure 3 and Figure 4 As shown, the image processing method includes the training process of the image recognition model and the inference process of the image recognition model. Figure 2 , Figure 4 Terminal 30a in the text can refer to Figure 1 Any terminal in the terminal cluster, Figure 2 , Figure 3 as well as Figure 4 Server 31a in the text can refer to Figure 1Server 10 in the middle. For example... Figure 2 As shown, the training process of the image recognition model includes the following steps S21 to S22:

[0040] S21. Train the initial image recognition model to obtain the target image recognition model. For details, please refer to [link to documentation]. Figure 3 As shown, server 31a acquires a training dataset, which includes K sample images and a labeled road attribute for each sample image, where K is a positive integer greater than or equal to 1. Here, a sample image can refer to an image captured of a driving road, a driving road can refer to a sample object in the sample image, and a labeled road attribute can refer to the labeled object attribute of the sample object. The labeled road attribute can refer to one or more of the following: driving road confidence score, vehicle information, lane information, traffic sign information, pedestrian information, etc. Vehicle information can include vehicle type, size, location, etc. Lane information can include lane lines, etc. Traffic sign information includes traffic signs, traffic lights, etc., and pedestrian information includes the number of pedestrians in the driving road, etc. The driving road confidence score reflects the probability that a corresponding sample image contains a driving road; that is, the higher the probability that a sample image contains a driving road, the higher the confidence score of the driving road in that sample image, and the lower the probability that a sample image contains a driving road, the lower the confidence score of the driving road in that sample image. Road attribute annotation can refer to the annotation obtained by manually annotating sample images. The road attribute annotations corresponding to sample image 1, sample image 2, ..., sample image K are respectively road attribute annotation 1, road attribute annotation 2, ..., road attribute annotation K.

[0041] Understandably, the K sample images here can refer to images taken at different times on the same road, or the K sample images can refer to images taken at the same time on different roads, or the K sample images can refer to images taken at the same time from different perspectives on the same road.

[0042] Server 31a can obtain the road attributes of driving roads in K sample image data through an initial image recognition model. Specifically, server 31a inputs sample image 1 from the training dataset into the initial image recognition model, and performs object prediction on sample image 1 to obtain the predicted road attribute 1 of the driving road in sample image 1. The predicted road attribute 1 can include one or more of the following: vehicle information, lane information, traffic sign information, and pedestrian information of the driving road in sample image 1. Server 31a inputs sample image 2 from the training dataset into the initial image recognition model, and performs object prediction on sample image 2 to obtain the predicted road attribute 2 of the driving road in sample image 2. The predicted road attribute 2 can include one or more of the following: vehicle information, lane information, traffic sign information, and pedestrian information of the driving road in sample image 2. And so on, server 31a can obtain the predicted road attributes corresponding to each of the K sample images.

[0043] Furthermore, after server 31a obtains the labeled road attributes and predicted road attributes corresponding to the K sample images, it can determine the object recognition error of the initial image recognition model based on the labeled road attributes and predicted road attributes corresponding to the K sample images. Based on this object recognition error, the initial image recognition model is adjusted to obtain a target image recognition model for recognizing the target image. Here, the target image can refer to an image taken of the target driving road.

[0044] S22. The target image recognition model is divided into terminal 30a and server 31a. For details, please refer to [link / reference]. Figure 4 As shown, to reduce the pressure on terminal 30a to deploy the target image recognition model, and to avoid terminal 30a uploading the original target image to server 31a, server 31a can collaboratively deploy the target image recognition model with terminal 30a. Specifically, the target image recognition model includes M network processing layers, including an image input layer, a recognition result output layer, and other processing layers. Server 30a can obtain the degree of abstraction of the image features in the target image by each of the M network processing layers. The degree of abstraction reflects the difficulty for the user to distinguish the road attributes of the driving road in the target image from the image features output by the network processing layer. That is, the higher the degree of abstraction of the network processing layer, the more difficult it is for the user to distinguish the road attributes of the driving road in the target image from the image features output by the network processing layer; conversely, the lower the degree of abstraction of the network processing layer, the easier it is for the user to distinguish the road attributes of the driving road in the target image from the image features output by the network processing layer.

[0045] Furthermore, server 31a can divide the M network processing layers according to their respective levels of abstraction to obtain a first image recognition sub-model and a second image recognition sub-model. The first image recognition sub-model is used to extract the low-level feature map of the target image, and the second image recognition sub-model is used to perform object recognition based on the low-level feature map to obtain the road attributes of the target driving road in the target image. The sum of the number of network processing layers in the first image recognition sub-model and the second image recognition sub-model is M. Assuming the first image recognition sub-model has F network processing layers and the second image recognition sub-model has E network processing layers, with E and F being positive integers greater than 1, the F network processing layers in the first image recognition sub-model include an image input layer and F-1 network processing layers adjacent to the image input layer. The E network processing layers in the second image recognition sub-model include a recognition result output layer and E-1 network processing layers adjacent to the recognition result output layer. The aforementioned low-level feature map can refer to image features in the target image whose level of abstraction is greater than a threshold, meaning that the user cannot distinguish the road attributes of the target driving road in the target image from this low-level feature map. After obtaining the first image recognition sub-model and the second image recognition sub-model, server 31a can deploy the first image recognition sub-model in terminal 30a and deploy the second image recognition sub-model in server 31a.

[0046] Furthermore, the recommendation process of the image recognition model includes the following steps S23 to S26:

[0047] S23, Terminal 30a extracts the low-level feature map of the target image.

[0048] S24. Terminal 30a uploads the underlying feature map to server 31a.

[0049] S25, Server 31a performs inference based on the underlying feature map.

[0050] S26. Server 31a returns the inference result to terminal 30a.

[0051] In steps S23 to S26, as follows Figure 4As shown, when it is necessary to identify the road attributes of a target road in a target image, terminal 30a can input the target image into a first image recognition sub-model. The first image recognition sub-model extracts low-level features from the target image to obtain a low-level feature map, which terminal 30a can then send to server 31a. Server 31a can receive the low-level feature map sent by terminal 30a, input it into a second image recognition sub-model, and perform object recognition on the low-level feature map to obtain the road attributes of the target road. The server 31a then returns the road attributes of the target road to terminal 30a. In other words, terminal 30a and server 31a collaborate to implement the inference process of the target image recognition model. After server 31a obtains the inference result, it returns the inference result (i.e., the road attributes) to terminal 30a. After receiving the road attributes of the target road, terminal 30a can perform desensitization processing on the target image, such as masking pedestrians in the target image.

[0052] It is understood that the aforementioned first image recognition sub-model can be deployed on multiple terminals, enabling multiple terminals to share the same second image recognition sub-model, which is beneficial for improving the utilization rate of the target image recognition model. Meanwhile, the server executing the training process of the aforementioned initial image recognition model and the server used to deploy the second image recognition sub-model can be the same server or different servers. In particular, the training process of the aforementioned initial image recognition model can also be executed by the terminal itself; this application does not limit this.

[0053] In summary, by collaboratively deploying the target image recognition model on both the server and the terminal, it is not necessary to deploy all network processing layers of the target recognition model on the terminal, thus reducing the image processing burden on the terminal. When the terminal needs to identify the road attributes of a target road in a target image, the terminal only needs to upload the low-level feature map to the server. The server then uses the second image recognition sub-model to perform object recognition on the low-level feature map to obtain the road attributes of the target road. The terminal does not need to upload the original target image to the server, which avoids the leakage of privacy data in the target image and improves image recognition accuracy.

[0054] Further, please see Figure 5 This is a flowchart illustrating an image data processing method provided in an embodiment of this application. Figure 5 As shown, this method can be derived from... Figure 1 It can be executed by the terminal in the middle, or by Figure 1 The server in the middle can be used to execute it, or it can be executed by... Figure 1 The method is executed jointly by a terminal and a server. The device used to execute this method in this application can be collectively referred to as a computer device. The image data processing method may include the following steps:

[0055] S101. Obtain the target image recognition model for recognizing the target image; the target image recognition model includes M network processing layers.

[0056] In this application, the amount of computational resources required for an image recognition model to run is generally related to its recognition accuracy. For example, the more computational resources an image recognition model requires, the higher its recognition accuracy and the more network processing layers it has. Conversely, the less computational resources an image recognition model requires, the lower its recognition accuracy and the fewer its network processing layers. Therefore, to improve image recognition accuracy, the target image recognition model here can refer to a non-lightweight image recognition model. A non-lightweight image recognition model can be one whose runtime computational resource requirements exceed a resource threshold. The target image recognition model (or the number of network processing layers) can specifically be determined based on the computational resources of the terminal and server used for collaborative deployment of the target image recognition model. The target image recognition model may include M network processing layers, where "network processing layer" is a collective term for all processing layers of the target image recognition model. The M network processing layers include an image input layer, a recognition result output layer, and other processing layers.

[0057] S102. Determine the degree of abstraction of the image features in the target image by each of the M network processing layers.

[0058] In this application, the computer device can determine the level of abstraction corresponding to the M network processing layers using either of the following two methods: Method 1: Generally, the network processing layer farther away from the image output layer among the M network processing layers has a higher level of abstraction of the image features in the target image; conversely, the network processing layer closer to the image output layer among the M network processing layers has a lower level of abstraction of the image features in the target image. Therefore, the computer device can determine the level of abstraction of the image features in the target image by each of the M network processing layers based on their positions in the target image recognition model. Method 2: The computer device can obtain a test image and the labeled object attributes of the objects in the test image. The test image refers to an image used to test the level of abstraction corresponding to each of the M network processing layers. The test image is input into the target image recognition model, and the output results of each of the M network processing layers for the test image are obtained. Based on the sum of the output results of the M network processing layers, the difficulty of distinguishing the target object in the target image from the output results is determined. Based on this difficulty, the level of abstraction of the image features in the target image by each of the M network processing layers is determined. The aforementioned determination of the difficulty of distinguishing target objects in the target image from the output results includes: inputting the output results corresponding to M network processing layers into the same lightweight image recognition model; using the lightweight image recognition model to identify test objects from the output results corresponding to the M network processing layers to obtain M test object attributes; determining the object recognition error of the lightweight image recognition model based on the M test object attributes and the labeled object attributes; and determining the difficulty of distinguishing target objects in the target image from the output results based on the object recognition error.

[0059] Understandably, the aforementioned M network processing layers include an image input layer and a recognition result output layer. The method for determining the level of abstraction of the image features in the target image by each of the M network processing layers can include: Method 1: The computer device can generate layer numbers corresponding to the M network processing layers in an incremental manner, from the image input layer to the recognition result output layer. For example, the layer number of the image input layer is 1, the layer number of the network processing layer following the image input layer is 2, ..., and the layer number of the recognition result output layer is M. The layer number reflects the position of the network processing layer in the target image recognition model. Based on the layer numbers corresponding to the M network processing layers, the level of abstraction of the image features in the target image by each of the M network processing layers is determined. At this point, there is a positive correlation between the layer number of the network processing layer Pi and the degree of abstraction of the image features in the target image by the network processing layer Pi, where i is a positive integer less than or equal to M; that is, the larger the layer number of the network processing layer Pi, the higher the degree of abstraction of the image features in the target image by the network processing layer Pi; conversely, the smaller the layer number of the network processing layer Pi, the lower the degree of abstraction of the image features in the target image by the network processing layer Pi.

[0060] Method 2: The computer device can generate M network processing layers with decreasing layer numbers in the direction from the image input layer to the recognition result output layer. For example, the layer number of the image input layer is M, the layer number of the network processing layer after the image input layer is M-1, ..., and the layer number of the recognition result output layer is 1. Based on the layer numbers of these M network processing layers, the degree of abstraction of the image features in the target image by each of the M network processing layers is determined. In this case, there is a negative correlation between the layer number of network processing layer Pi and the degree of abstraction of the image features in the target image by network processing layer Pi, where i is a positive integer less than or equal to M; that is, the larger the layer number of network processing layer Pi, the lower the degree of abstraction of the image features in the target image by network processing layer Pi; conversely, the smaller the layer number of network processing layer Pi, the higher the degree of abstraction of the image features in the target image by network processing layer Pi.

[0061] S103. Based on the level of abstraction corresponding to the M network processing layers, the M network processing layers are divided to obtain the first image recognition sub-model and the second image recognition sub-model.

[0062] In this application, a computer device can divide the M network processing layers according to their respective levels of abstraction to obtain a first image recognition sub-model and a second image recognition sub-model. The first image recognition sub-model is used to extract the low-level feature map of the target image, and the second image recognition sub-model is used to perform object recognition based on the low-level feature map to obtain the object attributes of the target object in the target image. The low-level feature map can be composed of image features in the target image that cannot distinguish the target object in the target image, and the target image cannot be reconstructed based on the low-level feature map. That is, the low-level feature map can be composed of image features in the target image whose level of abstraction is greater than an abstraction threshold.

[0063] Understandably, the aforementioned low-level feature map can be composed of image features from the target image that cannot distinguish the target object in the target image, which has the following implications: First, the user cannot distinguish the target object in the target image from the low-level feature map, meaning the target object in the target image is invisible in the low-level feature map, and the user cannot directly see the target object in the target image from the low-level feature map; Second, other image recognition models cannot identify the target object or the object attributes of the target object in the target image based on the low-level feature map. Other image recognition models refer to image recognition models other than the second image recognition sub-model. The reasons why the target object in the target image cannot be distinguished from the low-level feature map are as follows: 1. Because the low-level feature map is obtained by abstracting the image features in the target image through the first image recognition sub-model; 2. Because the first image recognition sub-model and the second image recognition sub-model are derived from the target image recognition model, it is equivalent to a matching relationship between the first image recognition sub-model and the second image recognition sub-model, that is, the output result of the first image recognition sub-model (i.e., the low-level feature map) can be understood by the second image recognition sub-model and further processed (such as object recognition). The low-level feature map is equivalent to an intermediate processing result in the image recognition process. Since other image recognition models do not know which stage the low-level feature map is the processing result of, they cannot understand the low-level feature map and cannot further process it. In other words, other image recognition models cannot identify (distinguish) the target object in the target image from the low-level feature map.

[0064] Understandably, the outputs of all M network processing layers can be considered image features of the target image. The image features output by different network processing layers all contain all the image features of the target image, but the level of abstraction of the image features output by different network processing layers varies. For example, the M network processing layers include network processing layer 1 and network processing layer 2. Network processing layer 1 and network processing layer 2 are adjacent in the target image recognition model; that is, the output of network processing layer 1 is the input of network processing layer 2, and the output of network processing layer 2 is obtained by further abstracting the output of network processing layer 1 (such as through convolution). Therefore, the level of abstraction of the image features output by network processing layer 2 is higher than that output by network processing layer 1. Furthermore, the aforementioned low-level feature map is composed of all image features in the target image with an abstraction level greater than a threshold, meaning the low-level feature map can reflect all the image features in the target image, thus ensuring image recognition accuracy.

[0065] Understandably, taking the example of a positive correlation between the layer number of any of the M network processing layers and the degree of abstraction of the corresponding network processing layer for the image features in the target image, the implementation method for dividing the M network processing layers is explained. Optionally, the implementation method for dividing the M network processing layers can adopt either of the following two methods: Method 1: The computer device can sequentially traverse the abstraction levels corresponding to the M network processing layers in ascending order of their respective layer numbers. When the abstraction level corresponding to the network processing layer Pa is greater than the abstraction level threshold for the first time, the network processing layers with layer numbers less than those of the network processing layer Pa, along with the network processing layer Pa itself, are determined as the first image recognition sub-model. The network processing layers other than those in the first image recognition sub-model are determined as the second image recognition sub-model, where a is a positive integer less than M. By adaptively dividing the target image recognition model according to the level of abstraction corresponding to the network processing layer, a first image recognition sub-model and a second image recognition sub-model are obtained. This avoids uploading the original target image to the server, which could lead to the leakage of privacy data in the target image and improves the security of image recognition processing.

[0066] Method 2: Due to the limited available computing resources in the terminal, to reduce the image recognition load, as few network processing layers as possible can be deployed, provided that the low-level feature maps output by some image recognition models deployed on the terminal cannot distinguish the target object. Specifically, the computer device can obtain the terminal's image processing performance parameters. Based on these parameters, N network processing layers that meet the terminal's image processing conditions are selected from the M network processing layers. These N layers have adjacent and continuous layer numbers, meaning they include the image input layer and N-1 adjacent network processing layers. Here, the terminal's image processing conditions refer to the terminal's ability to provide the computing resources required to run a maximum of H network processing layers, where N is less than or equal to H. Further, the abstraction levels of the N network processing layers are sequentially traversed according to their layer numbers from largest to smallest. If a network processing layer with an abstraction level greater than a threshold is found, it is identified as the first image recognition sub-model. The remaining network processing layers are designated as the second image recognition sub-model; these remaining network processing layers are all network processing layers other than those in the first image recognition sub-model. By adaptively dividing the target image recognition model according to the abstraction level of the network processing layers and the terminal's image processing performance parameters, the first and second image recognition sub-models are obtained. This avoids uploading the original target image to the server, preventing the leakage of privacy data in the target image, improving the security of image recognition processing, and reducing the image recognition load on the terminal.

[0067] Understandably, the process of selecting N network processing layers from the M network processing layers that meet the image processing conditions of the terminal, based on the terminal's image processing performance parameters, includes: the computer device obtaining the amount of computing resources required by the terminal for each of the M network processing layers. The computing resources corresponding to a network processing layer refer to the number of data items that the terminal needs to process per unit time. Further, based on the terminal's image processing performance parameters, the limited computing resources corresponding to the usable computing resources in the terminal are determined. That is, the limited computing resources are the usable computing resources in the terminal, or the limited computing resources refer to the number of data items that the terminal can process per unit time. Then, according to the ascending order of the layer numbers corresponding to the M network processing layers, the computing resources corresponding to the M network processing layers are accumulated sequentially. If the difference between the accumulated computing resources of N network processing layers and the limited computing resources is less than a difference threshold, then the accumulated N network processing layers are determined as the N network processing layers that meet the terminal's image processing conditions.

[0068] For example, assuming the computational resource limit is 50 lines and the difference threshold is 5 lines, the computational resource limits for network processing layer 1, network processing layer 2, network processing layer 3, ..., network processing layer M are 10 lines, 30 lines, 18 lines, ..., 20 lines, respectively. The computer device can calculate the sum of the computational resource limit for network processing layer 1 and network processing layer 2, obtaining a cumulative resource limit 1 of 40 lines. The difference between cumulative resource limit 1 and the limit is 10 lines. Since the difference between cumulative resource limit 1 and the limit is greater than the difference threshold, the network processing layers are traversed again to calculate the sum of cumulative resource limit 1 and the computational resource limit for network processing layer 3, obtaining a cumulative resource limit 2 of 58 lines. The difference between cumulative resource limit 2 and the limit is 2 lines. Since the difference between cumulative resource limit 2 and the limit is less than the difference threshold, the traversal of the network processing layers is paused, and network processing layers 1, 2, and 3 are determined as the N network processing layers that meet the image processing conditions of this terminal.

[0069] S104. Deploy the first image recognition sub-model in a terminal used for collaborative deployment of the target image recognition model, and deploy the second image recognition sub-model in a server used for collaborative deployment of the target image recognition model; the terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image, and the server is used to obtain the low-level feature map from the terminal, call the second image recognition sub-model to perform object recognition on the low-level feature map, and obtain the object attributes of the target object in the target image.

[0070] In this application, after acquiring the first image recognition sub-model and the second image recognition sub-model, the computer device can deploy the first image recognition sub-model on the terminal, that is, send the first image recognition sub-model to the terminal and set up a runtime environment for running the first image recognition sub-model on the terminal. The second image recognition sub-model is then sent to the server, and a runtime environment for running the second image recognition sub-model is set up on the server, thereby achieving collaborative deployment of the target image recognition model. When it is necessary to recognize the target image, the terminal only needs to upload the underlying feature map to the server, and the server uses the second image recognition sub-model to perform object recognition on the underlying feature map to obtain the road attributes of the target road. The terminal does not need to upload the original target image to the server, which avoids the leakage of privacy data in the target image and improves image recognition accuracy.

[0071] In this application, the target image recognition model is adaptively divided based on the level of abstraction of image features in the target image across its various network processing layers. This allows for the deployment of a portion of the network processing layers (i.e., the first image recognition sub-model) on the terminal and another portion (i.e., the second image recognition sub-model) on the server, achieving collaborative deployment of the target image recognition model. This eliminates the need to deploy all network processing layers on the terminal, reducing the image processing load on the terminal. The level of abstraction here reflects the difficulty for a user to distinguish the target object (or object attributes) in the target image from the image features output by the network processing layer. A higher level of abstraction makes it more difficult for the user to distinguish the target object from the image features output by the network processing layer; conversely, a lower level of abstraction makes it easier for the user to distinguish the target object from the image features output by the network processing layer. The first image recognition sub-model, based on this level of abstraction, outputs a low-level feature map of the target image. The target object (or the object attribute of the target object) in the target image cannot be distinguished from this low-level feature map. Therefore, when the terminal needs to identify the object attribute of the target object in the target image, the terminal only needs to upload the low-level feature map to the server. The server then uses the second image recognition sub-model to perform object recognition on the low-level feature map and obtain the object attribute of the target object. The terminal does not need to upload the original target image to the server. This can avoid the leakage of privacy data in the target image and improve the accuracy of image recognition.

[0072] Further, please see Figure 6 This is a flowchart illustrating an image data processing method provided in an embodiment of this application. Figure 6 As shown, this method can be derived from... Figure 1 It can be executed by the terminal in the middle, or by Figure 1 The server in the middle can be used to execute it, or it can be executed by... Figure 1 The method is executed jointly by a terminal and a server. The device used to execute this method in this application can be collectively referred to as a computer device. The image data processing method may include the following steps:

[0073] S201. Obtain the annotation object attributes of the sample objects in the sample image.

[0074] In this application, the computer device can acquire sample images and labeled object attributes of the sample images according to the application scenario of the target image recognition model. For example, when the target image recognition model is used to identify the road attributes of a driving road, the sample image can refer to an image obtained by taking a picture of the driving road, the sample object in the sample image can refer to the driving road, and the labeled object attributes of the sample object can refer to road attributes, including vehicle information, pedestrian information, traffic light information, etc., on the driving road. As another example, when the target image recognition model is used to recognize a face, the sample object can refer to an image including a person, the sample object can refer to a person, and the labeled object attributes of the sample object can refer to the user's basic information about the sample object.

[0075] S202. Call the initial image recognition model to perform object prediction on the above sample image and obtain the predicted object attributes of the sample object.

[0076] In this application, a computer device can input the sample image into an initial image recognition model, and use the initial image recognition model to predict the object in the sample image to obtain the predicted object attributes of the sample object.

[0077] S203. Based on the predicted object attribute and the labeled object attribute, adjust the initial image recognition model to obtain a target image recognition model for recognizing the target image.

[0078] In this application, if the predicted object attributes are closer to the labeled object attributes, it indicates that the object recognition error of the initial image recognition model is relatively low; conversely, if the predicted object attributes differ significantly from the labeled object attributes, it indicates that the object recognition error of the initial image recognition model is relatively high. Therefore, the computer device can determine the object recognition error of the initial image recognition model based on the predicted object attributes and the labeled object attributes, and adjust the initial image recognition model according to the object recognition error to obtain a target image recognition model for recognizing the target image.

[0079] It is understood that step S203 above includes: such as Figure 7As shown, for each of the K sample images, the computer device can determine the object recognition error of the initial image recognition model based on the predicted object attribute and the labeled object attribute, and determine the state of the initial image recognition model based on the object recognition error. If the object recognition error is greater than the error threshold, the initial image recognition model is determined to be in a non-converged state; if the object recognition error is less than or equal to the error threshold, the initial image recognition model is determined to be in a converged state. A converged state means that the object recognition error of the initial image recognition model has reached its minimum value or is close to its minimum value. A non-converged state means that the object recognition error of the initial image recognition model is not at its minimum value, or the difference between it and the minimum value is relatively large. Furthermore, if the initial image recognition model is in a converged state, it can be determined as the target image recognition model. If the initial image recognition model is not converged, its parameters are adjusted based on the object recognition error. The adjusted model is then trained using sample images until it converges. This adjusted model is then designated as the target image recognition model for recognizing the target image. By adjusting the initial image recognition model based on the predicted and labeled object attributes, the target image recognition model can be obtained, thus improving its image recognition accuracy.

[0080] Understandably, the labeled object attributes include the labeled object location, labeled object category, labeled object size, and labeled confidence level of the sample object; the predicted object attributes include the predicted object location, predicted object category, predicted object size, and predicted confidence level of the sample object; the labeled object location and the predicted object location are both used to reflect the location of the sample object in the sample image; the labeled confidence level and the predicted confidence level are both used to reflect the probability that the sample object exists in the sample image; the labeled confidence level and the predicted confidence level can be 0 or 1, where 0 indicates that the sample object does not exist in the sample image, and 1 indicates that the sample object exists in the sample image.

[0081] Understandably, the object recognition error of the initial image recognition model, determined based on the predicted object attribute and the labeled object attribute, includes: such as Figure 7As shown, for each of the K sample images, the computer device can use the labeled object position and the predicted object position to determine the object position recognition error of the initial image recognition model. This object position recognition error reflects the accuracy of the initial image recognition model in recognizing the position of the sample object in that sample image. It can also use the labeled object category and the predicted object category to determine the object category recognition error of the initial image recognition model. This object category recognition error reflects the accuracy of the initial image recognition model in recognizing the category of the sample object. Furthermore, it can use the labeled object size and the predicted object size to determine the object size recognition error of the initial image recognition model. This object size recognition error reflects the accuracy of the initial image recognition model in recognizing the size of the sample object. Finally, it can use the labeled confidence score and the predicted confidence score to determine the confidence score recognition error of the initial image recognition model. This confidence score recognition error reflects the accuracy of the initial image recognition model in predicting the confidence score. Then, based on the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error, the object recognition error of the initial image recognition model is determined. Specifically, this process is repeated to obtain the object recognition errors of the initial image recognition model for each of the K sample images. The cumulative sum of the object recognition errors corresponding to the K sample objects is determined as the object recognition error of the initial image recognition model. By measuring the object recognition error of the initial image recognition model from multiple perspectives such as the category, size, and location of the sample objects, the accuracy of determining the object recognition error of the initial image recognition model is improved, thereby improving the image recognition accuracy of the target image recognition model.

[0082] It is understood that the above-mentioned labeled object position includes the first coordinate value of the center point of the anchor box of the above-mentioned sample object in the first direction and the second coordinate value in the second direction; the above-mentioned predicted object position includes the third coordinate value of the center point of the predicted box of the above-mentioned sample object in the first direction and the fourth coordinate value in the second direction; the first direction may be the horizontal direction of the sample image, the second direction may be the vertical direction of the sample image, and the anchor box and the predicted box are rectangular boxes used to indicate the position of the sample object in the sample image.

[0083] The above-mentioned method of determining the object position recognition error of the initial image recognition model by using labeled object position and predicted object position includes: the computer device can obtain the difference between the first coordinate value and the third coordinate value as the first coordinate difference, which reflects the position recognition error of the initial image recognition model in the first direction; then, the difference between the second coordinate value and the fourth coordinate value is obtained as the second coordinate difference, which reflects the position recognition error of the initial image recognition model in the second direction. Further, the object position recognition error of the initial image recognition model is determined based on the first coordinate difference and the second coordinate difference; by statistically analyzing the position recognition errors of the initial image recognition model in multiple directions, the object position recognition error of the initial image recognition model is determined, thereby improving the accuracy of determining the position recognition error of the initial image recognition model.

[0084] It is understood that the aforementioned labeled object size includes the width and height of the anchor box, and the aforementioned predicted object size includes the width and height of the predicted box. The determination of the object size recognition error of the initial image recognition model using the labeled object size and the predicted object size includes: the computer device acquiring the difference between the width of the anchor box and the width of the predicted box as a width deviation, which reflects the width recognition error of the initial image recognition model for the sample object; acquiring the difference between the height of the anchor box and the height of the predicted box as a height deviation, which reflects the height recognition error of the initial image recognition model for the sample object; and determining the object size recognition error of the initial image recognition model based on the width deviation and the height deviation. By statistically analyzing the height recognition error and height recognition error of the initial image recognition model for the sample object, the object size recognition error of the initial image recognition model is determined, thereby improving the accuracy of determining the object size recognition error of the initial image recognition model.

[0085] Understandably, determining the object recognition error of the initial image recognition model based on the aforementioned object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error includes: the computer device being able to acquire the application scenario of the target image recognition model; and, based on the application scenario of the target image recognition model, determining the level of attention given to the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error in the aforementioned application scenario. For example, in a vehicle-to-everything (V2X) scenario, greater attention is paid to whether the sample object is a vehicle or a pedestrian, and the position of the sample object on the road. Therefore, in a V2X scenario, greater attention is paid to the object category recognition error and object location recognition error of the initial image recognition model. Similarly, in a face recognition scenario, greater attention is paid to whether a face exists in the sample image, and whether the face belongs to the target population, etc. Therefore, in a face recognition scenario, greater attention is paid to the confidence level recognition error and object category recognition error of the initial image recognition model. Therefore, the computer device can determine the weights corresponding to the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error based on the level of attention. That is, the higher the level of attention corresponding to a certain recognition error, the greater its weight; conversely, the lower the level of attention corresponding to a certain recognition error, the lower its weight. Furthermore, using the weights corresponding to the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error, a weighted summation is performed on these errors to obtain the object recognition error of the initial image recognition model. By adaptively determining the level of attention corresponding to each dimension of the recognition error based on the application scenario corresponding to the target image recognition model, and then performing a weighted summation on the recognition errors of each dimension based on the level of attention, the object recognition error of the initial image recognition model is obtained, thus improving the accuracy of the object recognition error of the initial image recognition model.

[0086] For example, the object recognition error of this initial image recognition model can be expressed by the following formula (1):

[0087]

[0088] In formula (1), S represents the predicted bounding box, and B represents the number of anchor boxes in the sample image. This indicates whether the (i, j) position in the predicted sample image obtained by the initial image recognition model is within the sample object; the value is 1 if it exists, and 0 otherwise. ij This represents the coordinates of the center point of the predicted bounding box located at position (i, j) in the sample image along the first direction. This represents the coordinates of the center point of the anchor box located at position (i, j) in the sample image along the first direction, y ij This represents the coordinates of the center point of the predicted bounding box located at position (i, j) in the sample image along the second direction. This represents the coordinates of the center point of the anchor box at position (i, j) in the sample image along the second direction. wij represents the width of the predicted box at position (i, j) in the sample image. h represents the width of the anchor box located at position (i, j) in the sample image. ij This represents the height of the predicted bounding box located at position (i, j) in the sample image. This represents the height of the anchor box located at position (i, j) in the sample image. C ij This represents the prediction confidence that a sample object exists at position (i, j) in the sample image. p represents the confidence level of the label containing a sample object at position (i, j) in the sample image; i (k) represents the probability that the sample object located at position (i, j) in the predicted sample image belongs to the kth category; Let represent the probability that the sample object located at position (i, j) in the labeled sample image belongs to the k-th category, where k belongs to c, c is the category set, and k is the k-th category in the category set. In formula (1), the right side of the equal sign in the first row represents the object position recognition error of the initial image recognition model, the second row represents the object size recognition error of the initial image recognition model, the third row represents the confidence recognition error of the initial image recognition model, and the fourth row represents the object category recognition error of the initial image recognition model. The weights corresponding to the object position recognition error and the object size recognition error are both α, the weight corresponding to the confidence recognition error is β, and the weight corresponding to the object category recognition error is γ.

[0089] S204. Obtain the target image recognition model for recognizing the target image; the target image recognition model includes M network processing layers.

[0090] Understandably, target image recognition models can refer to convolutional neural network models, deep learning network models, etc. Figure 8 As shown, taking a target image recognition model as an example of a deep learning network model, this target image recognition model consists of at least two basic convolutional modules and at least two residual modules. The modules in the target image recognition model can be collectively referred to as network processing layers. Figure 8As shown, the target image recognition model includes a basic convolutional module 60a, a residual module 61a, a residual module 62a, a residual module 63a, a residual module 64a, a residual module 65a, a residual module 66a, a residual module 67a, and a basic convolutional module 68a. The basic convolutional modules 60a and 68a are the image input layer and the recognition result input layer of the target image recognition model, respectively, and each residual module is an intermediate network processing layer of the target image recognition model. The parameters of each residual module can be different. For example, the stride of residual modules 62a, 64a, and 66a is 2, while the stride of residual modules 61a, 63a, 65a, and 67a is 1. The resolution of the basic convolutional module 68a is 1 / 16 of the resolution of the basic convolutional module 60a. The number of channels of the basic convolutional module 68a is B*(4+1+c), where B is the number of anchor boxes, 4 represents the offset regression of the center x-coordinate, center y-coordinate, length, and width of each anchor box, 1 represents the confidence level of whether it is the target object, and c is the object category of the target object, which is 2 categories in this case. Therefore, the feature channels of the basic convolutional module 68a are B*7. Optionally, a residual module may consist of a basic convolutional layer 70b, a normalization layer 71b, a basic convolutional layer 72b, and a normalization layer 73b; a basic convolutional module may consist of a basic convolutional layer 80c and a normalization layer 81c; and the basic convolutional layers may be 3x3 convolutional networks. The residual module is primarily used for downsampling image features, the basic convolutional module 60a is used to extract image features from the target image, and the basic convolutional module 68a is primarily used to output the recognition result.

[0091] S205. Determine the degree of abstraction of the image features in the target image by each of the M network processing layers.

[0092] S206. Based on the level of abstraction corresponding to the M network processing layers, the M network processing layers are divided to obtain the first image recognition sub-model and the second image recognition sub-model.

[0093] S207. Deploy the first image recognition sub-model in a terminal used for collaborative deployment of the target image recognition model, and deploy the second image recognition sub-model in a server used for collaborative deployment of the target image recognition model; the terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image, and the server is used to obtain the low-level feature map from the terminal, call the second image recognition sub-model to perform object recognition on the low-level feature map, and obtain the object attributes of the target object in the target image.

[0094] It is understood that the explanation of step S204 in this application can be referred to the above. Figure 4The explanation of step S101 in the previous section and the explanation of step S205 in this application can be found above. Figure 4 The explanation of step S102 is provided above. The explanation of step S206 in this application can be found in the above explanation. Figure 4 The explanation of step S103 in the previous section, and the explanation of step S207 in this application, can be found above. Figure 4 The explanation of step S104 is repeated here, and will not be repeated here.

[0095] Understandably, when it is necessary to use a target image recognition model to identify the object attributes of a target object in a target image, and the aforementioned computer device refers to a terminal, the terminal can call the locally stored first image recognition sub-model to extract low-level features from the target image, obtaining a low-level feature map. The target object to be identified in the target image is invisible in the low-level feature map, and the target image cannot be recovered from the low-level feature map. The low-level feature map is sent to the server, which calls the aforementioned second image recognition sub-model in the server to perform object recognition on the low-level feature map, obtaining the object attributes of the target object in the target image. Further, the server receives the object attributes of the target object sent by the server, determines the privacy data in the target image based on the object attributes, and performs desensitization processing on the privacy data in the target image. Here, privacy data can refer to vehicle license plates, human eyes, etc. The terminal only needs to upload the low-level feature map to the server, and the server uses the second image recognition sub-model to perform object recognition on the low-level feature map to obtain the object attributes of the target object. The terminal does not need to upload the original target image to the server, thus avoiding the leakage of privacy data in the target image and improving image recognition accuracy.

[0096] In this application, an initial image recognition model is trained using sample images and the labeled object attributes of sample objects within those images to obtain a target image recognition model, thereby improving the image recognition accuracy of the target image recognition model. Furthermore, based on the degree of abstraction of image features in the target image by each network processing layer of the target image recognition model, the model is adaptively partitioned. This allows for the deployment of a portion of the network processing layers (i.e., the first image recognition sub-model) on the terminal and another portion (i.e., the second image recognition sub-model) on the server, achieving collaborative deployment of the target image recognition model. This eliminates the need to deploy all network processing layers of the target recognition model on the terminal, reducing the image processing load on the terminal. Here, the degree of abstraction reflects the difficulty for a user to distinguish the target object (or object attributes) in the target image from the image features output by the network processing layer. A higher degree of abstraction for a network processing layer makes it more difficult for the user to distinguish the target object from the image features output by the network processing layer; conversely, a lower degree of abstraction makes it easier for the user to distinguish the target object from the image features output by the network processing layer. The first image recognition sub-model, based on this level of abstraction, outputs a low-level feature map of the target image. The target object (or the object attribute of the target object) in the target image cannot be distinguished from this low-level feature map. Therefore, when the terminal needs to identify the object attribute of the target object in the target image, the terminal only needs to upload the low-level feature map to the server. The server then uses the second image recognition sub-model to perform object recognition on the low-level feature map and obtain the object attribute of the target object. The terminal does not need to upload the original target image to the server. This can avoid the leakage of privacy data in the target image and improve the accuracy of image recognition.

[0097] Please see Figure 9 This is a schematic diagram of the structure of an image data processing apparatus provided in an embodiment of this application. The aforementioned image data processing apparatus can be a computer program (including program code) running in a network device; for example, the image data processing apparatus is an application software. This apparatus can be used to execute the corresponding steps in the method provided in the embodiments of this application. Figure 9 As shown, the image data processing device may include: an acquisition module 911, a determination module 912, a division module 913, and a deployment module 914.

[0098] The acquisition module 911 is used to acquire the target image recognition model for recognizing the target image, and the image processing performance parameters of the terminal for collaboratively deploying the target image recognition model; the target image recognition model includes M network processing layers.

[0099] The determination module 912 is used to determine the degree of abstraction of the image features in the target image by the above M network processing layers respectively;

[0100] The partitioning module 913 is used to partition the M network processing layers according to the abstraction level of the M network processing layers and the image processing performance parameters of the terminal, so as to obtain the first image recognition sub-model and the second image recognition sub-model.

[0101] The deployment module 914 is used to deploy the first image recognition sub-model in the terminal and the second image recognition sub-model in the server used for collaborative deployment of the target image recognition model. The terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image. The server is used to obtain the low-level feature map from the terminal and call the second image recognition sub-model to perform object recognition on the low-level feature map to obtain the object attributes of the target object in the target image.

[0102] It is understood that the above M network processing layers include an image input layer and a recognition result output layer; the determination module 912 includes a generation unit 91a and a first determination unit 92a;

[0103] The generation unit 91a is used to generate the layer numbers corresponding to the M network processing layers in sequence according to the direction from the image input layer to the recognition result output layer;

[0104] The first determining unit 92a is used to determine the degree of abstraction of the image features in the target image by the M network processing layers according to the layer numbers corresponding to the M network processing layers.

[0105] It is understandable that there is a positive correlation between the layer number of the network processing layer in the above M network processing layers and the degree of abstraction of the image features in the above target image by the corresponding network processing layer; the partitioning module 913 includes an acquisition unit 93b, a filtering unit 94b, a traversal unit 95b and a second determination unit 96b.

[0106] The first acquisition unit 93b is used to acquire the image processing performance parameters of the aforementioned terminal.

[0107] The filtering unit 94b is used to filter out N network processing layers from the M network processing layers that meet the image processing conditions of the terminal according to the image processing performance parameters of the terminal; the layer numbers corresponding to the N network processing layers are adjacent and continuous.

[0108] Traversal unit 95b is used to traverse the abstraction levels corresponding to the above N network processing layers in descending order of their layer numbers.

[0109] The second determining unit 96b is used to determine the N network processing layers as the first image recognition sub-model if there is a network processing layer with a corresponding abstraction level greater than the abstraction level threshold among the N network processing layers traversed; and to determine the remaining network processing layers as the second image recognition sub-model; the remaining network processing layers are the network processing layers other than the network processing layers in the first image recognition sub-model among the M network processing layers.

[0110] Understandably, the filtering unit 94b selects N network processing layers from the M network processing layers that meet the image processing conditions of the terminal, based on the image processing performance parameters of the terminal, including:

[0111] The amount of computing resources required by the terminal to acquire the above M network processing layers;

[0112] Based on the image processing performance parameters of the aforementioned terminal, determine the limited amount of computing resources corresponding to the computing resources in the aforementioned terminal that are in a usable state;

[0113] According to the layer number corresponding to the above M network processing layers in ascending order, the computing resources corresponding to the above M network processing layers are accumulated sequentially.

[0114] If the difference between the cumulative computational resources of N network processing layers out of the above M network processing layers and the above-mentioned limited computational resources is less than the difference threshold, then the N network processing layers that have been accumulated are determined to be the N network processing layers that meet the above-mentioned image processing conditions of the terminal.

[0115] Understandably, the acquisition module 911 includes a second acquisition unit 96c, a prediction unit 97c, and an adjustment unit 98c:

[0116] The second acquisition unit 96c is used to acquire the labeled object attributes of sample objects in the sample image;

[0117] Prediction unit 97c is used to call the initial image recognition model to predict objects in the above sample images and obtain the predicted object attributes of the above sample objects.

[0118] The adjustment unit 98c is used to adjust the initial image recognition model according to the predicted object attributes and the labeled object attributes to obtain a target image recognition model for recognizing the target image.

[0119] Understandably, the adjustment unit 98c adjusts the initial image recognition model based on the predicted object attributes and the labeled object attributes to obtain a target image recognition model for recognizing the target image, including:

[0120] Based on the predicted object attributes and the labeled object attributes, the object recognition error of the initial image recognition model is determined.

[0121] Based on the object recognition error mentioned above, determine the state of the initial image recognition model.

[0122] If the initial image recognition model is in a non-converged state, the model parameters of the initial image recognition model are adjusted according to the object recognition error.

[0123] Once the adjusted initial image recognition model has converged, it will be determined as the target image recognition model for recognizing the aforementioned target images.

[0124] It is understood that the aforementioned labeled object attributes include the labeled object location, labeled object category, labeled object size, and labeled confidence level of the aforementioned sample object; the aforementioned predicted object attributes include the predicted object location, predicted object category, predicted object size, and predicted confidence level of the aforementioned sample object; the aforementioned labeled object location and the aforementioned predicted object location are both used to reflect the location of the aforementioned sample object in the aforementioned sample image; the aforementioned labeled confidence level and the aforementioned predicted confidence level are both used to reflect the probability that the aforementioned sample object exists in the aforementioned sample image;

[0125] The adjustment unit 98c determines the object recognition error of the initial image recognition model based on the predicted object attributes and the labeled object attributes, including:

[0126] Using the labeled object location and the predicted object location, the object location recognition error of the initial image recognition model is determined.

[0127] Using the above-mentioned labeled object categories and the above-mentioned predicted object categories, the object category recognition error of the above-mentioned initial image recognition model is determined;

[0128] Using the above-mentioned labeled object size and the above-mentioned predicted object size, the object size recognition error of the above-mentioned initial image recognition model is determined;

[0129] Using the above-mentioned labeled confidence and the above-mentioned predicted confidence, the confidence recognition error of the above-mentioned initial image recognition model is determined;

[0130] Based on the object location recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error, the object recognition error of the initial image recognition model is determined.

[0131] It is understood that the above-mentioned labeled object position includes the first coordinate value of the center point of the anchor box of the above-mentioned sample object in the first direction and the second coordinate value in the second direction; the above-mentioned predicted object position includes the third coordinate value of the center point of the predicted box of the above-mentioned sample object in the first direction and the fourth coordinate value in the second direction.

[0132] Adjustment unit 98c uses the labeled object position and the predicted object position to determine the object position recognition error of the initial image recognition model, including:

[0133] The difference between the first coordinate value and the third coordinate value is obtained and used as the first coordinate difference.

[0134] The difference between the second coordinate value and the fourth coordinate value is obtained and used as the second coordinate difference.

[0135] Based on the first coordinate difference and the second coordinate difference, the object position recognition error of the initial image recognition model is determined.

[0136] It is understood that the dimensions of the above-mentioned labeled objects include the width and height of the above-mentioned anchor boxes, and the dimensions of the above-mentioned predicted objects include the width and height of the above-mentioned predicted boxes;

[0137] Adjustment unit 98c uses the above-mentioned labeled object size and the above-mentioned predicted object size to determine the object size recognition error of the above-mentioned initial image recognition model, including:

[0138] The difference between the width of the anchor box and the width of the prediction box is obtained as the width deviation;

[0139] The difference between the height of the anchor box and the height of the predicted box is obtained as the height deviation;

[0140] Based on the aforementioned width deviation and height deviation, the object size recognition error of the initial image recognition model is determined.

[0141] Understandably, the adjustment unit 98c determines the object recognition error of the initial image recognition model based on the object location recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error, including:

[0142] Find application scenarios for the above target image recognition model;

[0143] Based on the application scenarios of the above target image recognition model, the attention to the above object position recognition error, the above object category recognition error, the above object size recognition error and the above confidence recognition error are determined respectively in the above application scenarios.

[0144] Based on the above level of attention, determine the weights corresponding to the above object location recognition error, the above object category recognition error, the above object size recognition error, and the above confidence recognition error, respectively.

[0145] By using the weights corresponding to the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error, respectively, a weighted summation is performed on the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error to obtain the object recognition error of the initial image recognition model.

[0146] Understandably, the above-mentioned device also includes a calling module 915, a sending module 916, and a receiving module 917;

[0147] Module 915 is invoked to call the first image recognition sub-model stored locally to extract the underlying features of the target image and obtain the underlying feature map.

[0148] The sending module 916 is used to send the aforementioned low-level feature map to the server. The server is used to call the aforementioned second image recognition sub-model in the server to perform object recognition on the aforementioned low-level feature map and obtain the object attributes of the target object in the aforementioned target image.

[0149] The receiving module 917 is used to receive the object attributes of the target object sent by the server, and to perform desensitization processing on the target image based on the object attributes.

[0150] According to one embodiment of this application, Figure 5 The steps involved in the image data processing method shown can be derived from... Figure 9 The image data processing apparatus shown is executed by various modules within it. For example, Figure 5 Step S101 shown can be performed by Figure 9 The acquisition module 911 in the middle is used to execute, Figure 5 Step S102 shown can be performed by Figure 9 The determination module 912 in the middle is used to execute; Figure 5 Step S103 shown can be performed by Figure 9 The partitioning module 913 in the middle is used for execution; Figure 5 Step S104 shown can be performed by Figure 9 The deployment module 914 in the middle is used to execute it.

[0151] According to one embodiment of this application, Figure 9The modules in the image data processing apparatus shown can be individually or entirely combined into one or more units, or some of these units can be further divided into at least two functionally smaller sub-units to achieve the same operation without affecting the technical effects of the embodiments of this application. The above modules are based on logical functional division. In practical applications, the function of one module can be implemented by at least two units, or the function of at least two modules can be implemented by one unit. In other embodiments of this application, the image data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by at least two units.

[0152] According to one embodiment of this application, a computer program (including program code) capable of performing the steps involved in the corresponding methods described above can be executed on a general-purpose computer device, such as a computer, which includes processing components and storage components such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), to construct a system as described above. Figure 9 The image data processing apparatus shown herein, and the image data processing method for implementing the embodiments of this application, are described. The computer program described above may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0153] In this application, the target image recognition model is adaptively divided based on the level of abstraction of image features in the target image across its various network processing layers. This allows for the deployment of a portion of the network processing layers (i.e., the first image recognition sub-model) on the terminal and another portion (i.e., the second image recognition sub-model) on the server, achieving collaborative deployment of the target image recognition model. This eliminates the need to deploy all network processing layers on the terminal, reducing the image processing load on the terminal. The level of abstraction here reflects the difficulty for a user to distinguish the target object (or object attributes) in the target image from the image features output by the network processing layer. A higher level of abstraction makes it more difficult for the user to distinguish the target object from the image features output by the network processing layer; conversely, a lower level of abstraction makes it easier for the user to distinguish the target object from the image features output by the network processing layer. The first image recognition sub-model, based on this level of abstraction, outputs a low-level feature map of the target image. The target object (or the object attribute of the target object) in the target image cannot be distinguished from this low-level feature map. Therefore, when the terminal needs to identify the object attribute of the target object in the target image, the terminal only needs to upload the low-level feature map to the server. The server then uses the second image recognition sub-model to perform object recognition on the low-level feature map and obtain the object attribute of the target object. The terminal does not need to upload the original target image to the server. This can avoid the leakage of privacy data in the target image and improve the accuracy of image recognition.

[0154] Please see Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 10 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk drive. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 10As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0155] exist Figure 10 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0156] Obtain a target image recognition model for recognizing the target image; the target image recognition model includes M network processing layers;

[0157] Determine the degree of abstraction of the image features in the target image by each of the above M network processing layers;

[0158] Based on the level of abstraction corresponding to the above M network processing layers, the above M network processing layers are divided to obtain the first image recognition sub-model and the second image recognition sub-model;

[0159] The first image recognition sub-model is deployed in a terminal for collaborative deployment of the target image recognition model, and the second image recognition sub-model is deployed in a server for collaborative deployment of the target image recognition model. The terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image, and the server is used to obtain the low-level feature map from the terminal, call the second image recognition sub-model to perform object recognition on the low-level feature map, and obtain the object attributes of the target object in the target image.

[0160] Understandably, the aforementioned M network processing layers include an image input layer and a recognition result output layer; the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0161] Determine the level of abstraction of image features in the target image by each of the M network processing layers, including:

[0162] According to the direction from the image input layer to the recognition result output layer, the layer numbers corresponding to the above M network processing layers are generated sequentially.

[0163] Based on the layer numbers corresponding to the M network processing layers, the degree of abstraction of the image features in the target image by the M network processing layers is determined.

[0164] It is understandable that there is a positive correlation between the layer number of the M network processing layers and the degree of abstraction of the corresponding network processing layer for the image features in the target image; the processor 1001 can be used to call the device control application stored in the memory 1005 to divide the M network processing layers according to their respective degrees of abstraction, thereby obtaining a first image recognition sub-model and a second image recognition sub-model, including:

[0165] Obtain the image processing performance parameters of the aforementioned terminals;

[0166] Based on the image processing performance parameters of the aforementioned terminal, N network processing layers that meet the image processing conditions of the aforementioned terminal are selected from the M network processing layers; the layer numbers corresponding to the aforementioned N network processing layers are adjacent and continuous.

[0167] According to the layer number corresponding to the above N network processing layers in descending order, the abstraction level corresponding to the above N network processing layers is traversed in turn;

[0168] If, during the traversal of the above N network processing layers, there exists a network processing layer with an abstraction level greater than the abstraction level threshold, then the above N network processing layers are determined as the first image recognition sub-model.

[0169] The remaining network processing layers are determined as the second image recognition sub-model; the aforementioned remaining network processing layers are the network processing layers other than the network processing layers in the aforementioned first image recognition sub-model among the aforementioned M network processing layers.

[0170] Understandably, processor 1001 can be used to call the device control application stored in memory 1005 to select N network processing layers from the M network processing layers that meet the image processing conditions of the terminal, based on the image processing performance parameters of the terminal, including:

[0171] The amount of computing resources required by the terminal to acquire the above M network processing layers;

[0172] Based on the image processing performance parameters of the aforementioned terminal, determine the limited amount of computing resources corresponding to the computing resources in the aforementioned terminal that are in a usable state;

[0173] According to the layer number corresponding to the above M network processing layers in ascending order, the computing resources corresponding to the above M network processing layers are accumulated sequentially.

[0174] If the difference between the cumulative computational resources of N network processing layers out of the above M network processing layers and the above-mentioned limited computational resources is less than the difference threshold, then the N network processing layers that have been accumulated are determined to be the N network processing layers that meet the above-mentioned image processing conditions of the terminal.

[0175] Understandably, processor 1001 can be used to call the device control application stored in memory 1005 to acquire a target image recognition model for recognizing the target image, including:

[0176] Retrieve the labeled object attributes of sample objects in the sample image;

[0177] The initial image recognition model is invoked to predict objects in the above sample images, thereby obtaining the predicted object attributes of the above sample objects.

[0178] Based on the predicted object attributes and the labeled object attributes, the initial image recognition model is adjusted to obtain a target image recognition model for recognizing the target image.

[0179] Understandably, the processor 1001 can be used to call the device control application stored in the memory 1005 to adjust the initial image recognition model based on the predicted object attributes and the labeled object attributes, thereby obtaining a target image recognition model for recognizing the target image, including:

[0180] Based on the predicted object attributes and the labeled object attributes, the object recognition error of the initial image recognition model is determined.

[0181] Based on the object recognition error mentioned above, determine the state of the initial image recognition model.

[0182] If the initial image recognition model is in a non-converged state, the model parameters of the initial image recognition model are adjusted according to the object recognition error.

[0183] Once the adjusted initial image recognition model has converged, it will be determined as the target image recognition model for recognizing the aforementioned target images.

[0184] It is understood that the aforementioned labeled object attributes include the labeled object location, labeled object category, labeled object size, and labeled confidence level of the aforementioned sample object; the aforementioned predicted object attributes include the predicted object location, predicted object category, predicted object size, and predicted confidence level of the aforementioned sample object; the aforementioned labeled object location and the aforementioned predicted object location are both used to reflect the location of the aforementioned sample object in the aforementioned sample image; the aforementioned labeled confidence level and the aforementioned predicted confidence level are both used to reflect the probability that the aforementioned sample object exists in the aforementioned sample image;

[0185] Processor 1001 can be used to call the device control application stored in memory 1005 to determine the object recognition error of the initial image recognition model based on the predicted object attributes and the labeled object attributes, including:

[0186] Using the labeled object location and the predicted object location, the object location recognition error of the initial image recognition model is determined.

[0187] Using the above-mentioned labeled object categories and the above-mentioned predicted object categories, the object category recognition error of the above-mentioned initial image recognition model is determined;

[0188] Using the above-mentioned labeled object size and the above-mentioned predicted object size, the object size recognition error of the above-mentioned initial image recognition model is determined;

[0189] Using the above-mentioned labeled confidence and the above-mentioned predicted confidence, the confidence recognition error of the above-mentioned initial image recognition model is determined;

[0190] Based on the object location recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error, the object recognition error of the initial image recognition model is determined.

[0191] It is understood that the above-mentioned labeled object position includes the first coordinate value of the center point of the anchor box of the above-mentioned sample object in the first direction and the second coordinate value in the second direction; the above-mentioned predicted object position includes the third coordinate value of the center point of the predicted box of the above-mentioned sample object in the first direction and the fourth coordinate value in the second direction.

[0192] Processor 1001 can be used to call the device control application stored in memory 1005 to determine the object position recognition error of the initial image recognition model using the above-mentioned labeled object position and the above-mentioned predicted object position, including:

[0193] The difference between the first coordinate value and the third coordinate value is obtained and used as the first coordinate difference.

[0194] The difference between the second coordinate value and the fourth coordinate value is obtained and used as the second coordinate difference.

[0195] Based on the first coordinate difference and the second coordinate difference, the object position recognition error of the initial image recognition model is determined.

[0196] It is understood that the dimensions of the above-mentioned labeled objects include the width and height of the above-mentioned anchor boxes, and the dimensions of the above-mentioned predicted objects include the width and height of the above-mentioned predicted boxes;

[0197] Processor 1001 can be used to call the device control application stored in memory 1005 to determine the object size recognition error of the initial image recognition model using the above-mentioned labeled object size and the above-mentioned predicted object size, including:

[0198] The difference between the width of the anchor box and the width of the prediction box is obtained as the width deviation;

[0199] The difference between the height of the anchor box and the height of the predicted box is obtained as the height deviation;

[0200] Based on the aforementioned width deviation and height deviation, the object size recognition error of the initial image recognition model is determined.

[0201] Understandably, processor 1001 can be used to call the device control application stored in memory 1005 to determine the object recognition error of the initial image recognition model based on the object position recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error, including:

[0202] Find application scenarios for the above target image recognition model;

[0203] Based on the application scenarios of the above target image recognition model, the attention to the above object position recognition error, the above object category recognition error, the above object size recognition error and the above confidence recognition error are determined respectively in the above application scenarios.

[0204] Based on the above level of attention, determine the weights corresponding to the above object location recognition error, the above object category recognition error, the above object size recognition error, and the above confidence recognition error, respectively.

[0205] By using the weights corresponding to the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error, respectively, a weighted summation is performed on the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error to obtain the object recognition error of the initial image recognition model.

[0206] Understandably, processor 1001 can be used to call device control applications stored in memory 1005 to achieve:

[0207] The first image recognition sub-model stored locally is invoked to extract low-level features from the target image to obtain a low-level feature map.

[0208] The aforementioned low-level feature map is sent to the server, which then calls the aforementioned second image recognition sub-model in the server to perform object recognition on the aforementioned low-level feature map and obtain the object attributes of the target object in the aforementioned target image.

[0209] The system receives the object attributes of the target object sent by the server and performs desensitization processing on the target image based on the object attributes.

[0210] In this application, the target image recognition model is adaptively divided based on the level of abstraction of image features in the target image across its various network processing layers. This allows for the deployment of a portion of the network processing layers (i.e., the first image recognition sub-model) on the terminal and another portion (i.e., the second image recognition sub-model) on the server, achieving collaborative deployment of the target image recognition model. This eliminates the need to deploy all network processing layers on the terminal, reducing the image processing load on the terminal. The level of abstraction here reflects the difficulty for a user to distinguish the target object (or object attributes) in the target image from the image features output by the network processing layer. A higher level of abstraction makes it more difficult for the user to distinguish the target object from the image features output by the network processing layer; conversely, a lower level of abstraction makes it easier for the user to distinguish the target object from the image features output by the network processing layer. The first image recognition sub-model, based on this level of abstraction, outputs a low-level feature map of the target image. The target object (or the object attribute of the target object) in the target image cannot be distinguished from this low-level feature map. Therefore, when the terminal needs to identify the object attribute of the target object in the target image, the terminal only needs to upload the low-level feature map to the server. The server then uses the second image recognition sub-model to perform object recognition on the low-level feature map and obtain the object attribute of the target object. The terminal does not need to upload the original target image to the server. This can avoid the leakage of privacy data in the target image and improve the accuracy of image recognition.

[0211] It should be understood that the computer device 1000 described in the embodiments of this application can execute the image data processing method described in the corresponding embodiments above, and can also execute the methods described above. Figure 10 The description of the image data processing apparatus in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0212] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program executed by the aforementioned image data processing apparatus. This computer program includes program instructions, which, when executed by the processor, enable the execution of the image data processing method described in the corresponding embodiments above. Therefore, these descriptions will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments of this application, please refer to the description of the method embodiments of this application.

[0213] As an example, the above program instructions can be deployed and executed on a single computer device, or deployed and executed on at least two computer devices at a single location, or executed on at least two computer devices distributed across at least two locations and interconnected via a communication network. At least two computer devices distributed across at least two locations and interconnected via a communication network can form a blockchain network.

[0214] The aforementioned computer-readable storage medium can be the image data processing apparatus provided in any of the foregoing embodiments or the central storage unit of the aforementioned computer device, such as the hard disk or central storage of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both the central storage unit and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0215] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish content in different media, rather than to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0216] This application also provides a computer program product, including a computer program / instructions. When executed by a processor, the computer program / instructions implement the image data processing method described in the preceding embodiments; therefore, they will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0217] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0218] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable network-connected device to generate a machine, such that the instructions executable by the processor of the computer or other programmable network-connected device produce instructions for implementing the process. Figure 1 A flowchart or multiple flowcharts and / or structural diagrams Figure 1 The means for the functions specified in one or more boxes. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable network-connected device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means implemented in a process. Figure 1 One or more processes and / or structural diagrams Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable network-connected device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A flowchart or a structure diagram showing the steps of the function specified in one or more boxes.

[0219] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image data processing method, characterized in that, include: Obtain a target image recognition model for recognizing target images; the target image recognition model includes M network processing layers; the M network processing layers include an image input layer and a recognition result output layer; Obtain a test image and the labeled object attributes of the objects in the test image. Obtain the output results of the M network processing layers for the test image. Input the output results of the M network processing layers into the same lightweight image recognition model. Use the lightweight image recognition model to identify the test objects from the output results of the M network processing layers to obtain M test object attributes. Determine the M object recognition errors of the lightweight image recognition model based on the M test object attributes and the labeled object attributes. Determine the difficulty of distinguishing the target objects in the target image from the output results based on the M object recognition errors. Based on the difficulty levels corresponding to the M network processing layers, the degree of abstraction of the image features in the target image by each of the M network processing layers is determined. Following the direction from the image input layer to the recognition result output layer, the abstraction levels corresponding to the N network processing layers are sequentially processed; The layer numbers corresponding to the N network processing layers are adjacent and consecutive; If, during the traversal of the N network processing layers, there exists a network processing layer with an abstraction level greater than the abstraction level threshold, then the N network processing layers are determined as the first image recognition sub-model. The remaining network processing layers are determined as the second image recognition sub-model; the remaining network processing layers are the network processing layers other than the network processing layers in the first image recognition sub-model among the M network processing layers; The first image recognition sub-model is deployed on a terminal used for collaborative deployment of the target image recognition model, and the second image recognition sub-model is deployed on a server used for collaborative deployment of the target image recognition model. The terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image. The server is used to obtain the low-level feature map from the terminal, call the second image recognition sub-model to perform object recognition on the low-level feature map, and obtain the object attributes of the target object in the target image. The low-level feature map is composed of image features in the target image that cannot distinguish the target object, and the low-level feature map cannot reconstruct the target image.

2. The method as described in claim 1, characterized in that, The M network processing layers include an image input layer and a recognition result output layer; The method further includes: According to the direction from the image input layer to the recognition result output layer, the layer numbers corresponding to the M network processing layers are generated sequentially. Based on the layer numbers corresponding to the M network processing layers, the degree of abstraction of the image features in the target image by each of the M network processing layers is determined.

3. The method as described in claim 2, characterized in that, There is a positive correlation between the layer number of the network processing layer in the M network processing layers and the degree of abstraction of the image features in the target image by the corresponding network processing layer; The method further includes: Obtain the image processing performance parameters of the terminal; Based on the image processing performance parameters of the terminal, N network processing layers that meet the image processing conditions of the terminal are selected from the M network processing layers; the layer numbers corresponding to the N network processing layers are adjacent and continuous.

4. The method as described in claim 3, characterized in that, The step of selecting N network processing layers from the M network processing layers that meet the image processing conditions of the terminal based on the terminal's image processing performance parameters includes: To determine the amount of computing resources required by the terminal for each of the M network processing layers; Based on the image processing performance parameters of the terminal, determine the limited amount of computing resources corresponding to the computing resources in the terminal that are in a usable state; According to the M network processing layers, the computing resources corresponding to the M network processing layers are accumulated in ascending order of their respective layer numbers; If the difference between the cumulative computational resources of N network processing layers out of the M network processing layers and the limited computational resources is less than the difference threshold, then the N network processing layers that have been accumulated are determined as the N network processing layers that meet the image processing conditions of the terminal.

5. The method as described in claim 1, characterized in that, The step of obtaining a target image recognition model for recognizing the target image includes: Retrieve the labeled object attributes of sample objects in the sample image; The initial image recognition model is invoked to perform object prediction on the sample image, thereby obtaining the predicted object attributes of the sample object; Based on the predicted object attributes and the labeled object attributes, the initial image recognition model is adjusted to obtain a target image recognition model for recognizing the target image.

6. The method as described in claim 5, characterized in that, The step of adjusting the initial image recognition model based on the predicted object attributes and the labeled object attributes to obtain a target image recognition model for recognizing the target image includes: Based on the predicted object attributes and the labeled object attributes, the object recognition error of the initial image recognition model is determined; The state of the initial image recognition model is determined based on the object recognition error. If the initial image recognition model is in a non-converged state, the model parameters of the initial image recognition model are adjusted according to the object recognition error; Until the adjusted initial image recognition model is in a convergent state, the adjusted initial image recognition model is determined as the target image recognition model for recognizing the target image.

7. The method as described in claim 6, characterized in that, The labeled object attributes include the labeled object location, labeled object category, labeled object size, and labeled confidence level of the sample object; the predicted object attributes include the predicted object location, predicted object category, predicted object size, and predicted confidence level of the sample object; both the labeled object location and the predicted object location reflect the position of the sample object in the sample image; both the labeled confidence level and the predicted confidence level reflect the probability that the sample object exists in the sample image; The step of determining the object recognition error of the initial image recognition model based on the predicted object attributes and the labeled object attributes includes: The object position recognition error of the initial image recognition model is determined using the labeled object position and the predicted object position. The object category recognition error of the initial image recognition model is determined using the labeled object category and the predicted object category. The object size recognition error of the initial image recognition model is determined using the labeled object size and the predicted object size. The confidence recognition error of the initial image recognition model is determined using the labeled confidence and the predicted confidence. The object recognition error of the initial image recognition model is determined based on the object location recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error.

8. The method as described in claim 7, characterized in that, The labeled object position includes the first coordinate value of the center point of the anchor box of the sample object in the first direction and the second coordinate value in the second direction; the predicted object position includes the third coordinate value of the center point of the predicted box of the sample object in the first direction and the fourth coordinate value in the second direction. The step of determining the object location recognition error of the initial image recognition model using the labeled object location and the predicted object location includes: The difference between the first coordinate value and the third coordinate value is obtained as the first coordinate difference. The difference between the second coordinate value and the fourth coordinate value is obtained as the second coordinate difference. The object position recognition error of the initial image recognition model is determined based on the first coordinate difference and the second coordinate difference.

9. The method as described in claim 8, characterized in that, The dimensions of the labeled object include the width and height of the anchor box, and the dimensions of the predicted object include the width and height of the predicted box; The step of determining the object size recognition error of the initial image recognition model using the labeled object size and the predicted object size includes: The difference between the width of the anchor box and the width of the prediction box is obtained as the width deviation; The difference between the height of the anchor frame and the height of the prediction frame is obtained as the height deviation; The object size recognition error of the initial image recognition model is determined based on the width deviation and the height deviation.

10. The method as described in claim 7, characterized in that, Determining the object recognition error of the initial image recognition model based on the object location recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error includes: Obtain the application scenarios of the target image recognition model; Based on the application scenario of the target image recognition model, determine the attention to the object location recognition error, the object category recognition error, the object size recognition error, and the confidence recognition error respectively in the application scenario; Based on the level of attention, determine the weights corresponding to the object location recognition error, the object category recognition error, the object size recognition error, and the confidence level recognition error, respectively. The object recognition error of the initial image recognition model is obtained by weighting the object location recognition error, object category recognition error, object size recognition error, and confidence level recognition error according to their respective weights.

11. The method as described in claim 1, characterized in that, The method further includes: The first image recognition sub-model stored locally is invoked to extract low-level features from the target image to obtain a low-level feature map; The underlying feature map is sent to the server, which calls the second image recognition sub-model in the server to perform object recognition on the underlying feature map and obtain the object attributes of the target object in the target image. The system receives the object attributes of the target object sent by the server and performs desensitization processing on the target image based on the object attributes.

12. An image data processing apparatus, characterized in that, include: The acquisition module is used to acquire a target image recognition model for recognizing target images, and image processing performance parameters of a terminal for collaboratively deploying the target image recognition model; the target image recognition model includes M network processing layers; the M network processing layers include an image input layer and a recognition result output layer; The determination module is used to acquire a test image and the labeled object attributes of objects in the test image, acquire the output results of the M network processing layers for the test image, input the output results of the M network processing layers into the same lightweight image recognition model, perform test object recognition on the output results of the M network processing layers through the lightweight image recognition model to obtain M test object attributes, determine the M object recognition errors of the lightweight image recognition model based on the M test object attributes and the labeled object attributes, and determine the difficulty of distinguishing the target object in the target image from the output results based on the M object recognition errors. Based on the difficulty levels corresponding to the M network processing layers, the degree of abstraction of the image features in the target image by each of the M network processing layers is determined. The partitioning module is used to sequentially process the abstraction levels of N network processing layers according to the direction from the image input layer to the recognition result output layer; The layer numbers corresponding to the N network processing layers are adjacent and consecutive; If, during the traversal of the N network processing layers, there exists a network processing layer with an abstraction level greater than the abstraction level threshold, then the N network processing layers are determined as the first image recognition sub-model; the remaining network processing layers are determined as the second image recognition sub-model; the remaining network processing layers are the network processing layers other than those in the first image recognition sub-model among the M network processing layers. The deployment module is used to deploy the first image recognition sub-model in the terminal and the second image recognition sub-model in a server for collaborative deployment of the target image recognition model. The terminal is used to call the first image recognition sub-model to extract the low-level feature map of the target image. The server is used to obtain the low-level feature map from the terminal, call the second image recognition sub-model to perform object recognition on the low-level feature map, and obtain the object attributes of the target object in the target image. The low-level feature map is composed of image features in the target image that cannot distinguish the target object, and the low-level feature map cannot reconstruct the target image.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images