Image processing method, electronic device, storage medium and computer program product

Abstract

The embodiments of the present application provide an image processing method, electronic device, storage medium, and computer program product. The method includes: obtaining an image to be processed; performing image processing on the image to be processed using a trained student processing model; the model is obtained through the following training operations: obtaining a first sample image and corresponding first annotation information; inputting the first sample image into the teacher encoding module of the initial teacher processing model for encoding; obtaining a first initial teacher query feature; merging the first initial teacher query feature with the first real query feature; inputting the first teacher encoding feature and the merged first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding to obtain a first teacher processing result; training the initial teacher processing model based on at least the first teacher processing result and the first annotation information; and performing distillation training on the first initial student processing model using the trained teacher processing model. This method requires less training data.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically to an image processing method, electronic device, storage medium, and computer program product. Background Art

[0002] Image processing technologies such as 3D object detection and 3D instance segmentation are primarily performed using corresponding object detection and instance segmentation models. However, the performance of these models is often limited by the scale of data annotation and the size of the model. To mitigate this effect, existing techniques employ a training method that performs knowledge distillation on image processing models (object detection or instance segmentation models). Specifically, during the training phase, a teacher network with high performance and strong feature learning capabilities is used to assist a student network. The student network is then used for inference (or testing) during the inference phase (also known as the testing phase).

[0003] Knowledge distillation methods can improve the upper limit of detection performance. In existing technologies, for example, in object detection applications, a common distillation paradigm in object detection models involves distillation using data from different modalities. This data often includes 3D point cloud data. 3D point cloud data is large in volume and expensive to acquire, making such methods expensive to train and less practical. Therefore, a new image processing solution is needed to address these technical issues. Summary of the Invention

[0004] In view of the above problems, the present application is proposed. The present application provides an image processing method, an electronic device, a storage medium and a computer program product.

[0005] According to one aspect of the present application, an image processing method is provided, comprising: obtaining an image to be processed; performing image processing on the image to be processed using a trained student processing model to obtain a processing result of the image to be processed, wherein the image processing comprises three-dimensional target detection and / or three-dimensional instance segmentation, and the processing result comprises a target detection result and / or an instance segmentation result; the trained student processing model is obtained by training the following operations: obtaining a first sample image and corresponding first annotation information, wherein the first annotation information is used to indicate the three-dimensional position of the target object contained in the first sample image; inputting the first sample image into the teacher encoding module of the initial teacher processing model for encoding, and obtaining a first teacher encoding feature; obtaining a first initial teacher query feature, wherein the first initial teacher query feature comprises at least one At least one feature vector corresponding to a potential target object; merging a first initial teacher query feature with a first real query feature to obtain a first teacher query feature, wherein the first real query feature is obtained by position encoding based on the first annotation information; inputting the first teacher encoding feature and the first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding to obtain a first teacher processing result of the first sample image, the first teacher processing result including a target detection result and / or an instance segmentation result; training the initial teacher processing model based on at least the first teacher processing result and the first annotation information to obtain a trained teacher processing model; and performing distillation training on the first initial student processing model using the trained teacher processing model to obtain a trained student processing model.

[0006] Exemplarily, before using the trained teacher processing model to perform distillation training on the first initial student processing model to obtain the trained student processing model, the training operation also includes: obtaining a second sample image and corresponding second annotation information, the second annotation information is used to indicate the three-dimensional position of the target object contained in the second sample image; inputting the second sample image into the student encoding module of the second initial student processing model for encoding to obtain a first student encoding feature; obtaining a first student query feature, the first student query feature includes at least one feature vector corresponding one-to-one to at least one potential target object; inputting the first student encoding feature and the first student query feature into the student decoding module of the second initial student processing model for decoding to obtain a first student processing result of the second sample image, the first student processing result includes a target detection result and / or an instance segmentation result; training the second initial student processing model based at least on the first student processing result and the second annotation information to obtain the first initial student processing model.

[0007] Exemplarily, the teacher encoding module includes a feature extraction module and a position encoder module, and inputs the first sample image into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature, including: inputting the first sample image into the feature extraction module of the initial teacher processing model for feature extraction to obtain a first teacher image feature; obtaining a first position embedding feature corresponding to the first sample image; inputting the first teacher image feature and the first position embedding feature into the position encoder module of the initial teacher processing model for position encoding to obtain the first teacher encoding feature; the student encoding module includes a feature extraction module and a position encoder module, and inputting the second sample image into the student encoding module of the second initial student processing model for encoding to obtain a first student encoding feature, including: inputting the second sample image into the feature extraction module of the second initial student processing model for feature extraction to obtain a first student image feature; obtaining a second position embedding feature corresponding to the second sample image; inputting the first student image feature and the second position embedding feature into the position encoder module of the second initial student processing model for position encoding to obtain a first student encoding feature; wherein, the feature extraction module of the initial teacher processing model and the feature extraction module of the second initial student processing model are the same shared feature extraction module.

[0008] Exemplarily, the teacher decoding module includes a decoder module and a processing head, the processing head includes a detection head for outputting a target detection result and / or a segmentation head for outputting an instance segmentation result, the first teacher encoding feature and the first teacher query feature are input into the teacher decoding module of the initial teacher processing model for decoding, and the first teacher processing result of the first sample image is obtained, including: inputting the first teacher encoding feature and the first teacher query feature into the decoder module of the initial teacher processing model to obtain the first teacher decoding feature; inputting the first teacher decoding feature into the processing head of the initial teacher processing model to obtain the first teacher processing result; the student decoding module includes a decoder module and a processing head, processing The head includes a processing head of a teacher decoding module, including a detection head for outputting a target detection result and / or a segmentation head for outputting an instance segmentation result. The first student encoding feature and the first student query feature are input into the student decoding module of the second initial student processing model for decoding to obtain the first student processing result of the second sample image, including: inputting the first student encoding feature and the first student query feature into the decoder module of the second initial student processing model to obtain the first student decoding feature; inputting the first student decoding feature into the processing head of the second initial student processing model to obtain the first student processing result; wherein, the processing head of the initial teacher processing model and the processing head of the second initial student processing model are the same shared processing head.

[0009] Exemplarily, the second sample image is the same as the first sample image, wherein, in the process of training the initial teacher processing model based at least on the first teacher processing result and the first annotation information, the first prediction loss is determined based on the first teacher processing result and the first annotation information, and in the process of training the second initial student processing model based at least on the first student processing result and the second annotation information, the second prediction loss is determined based on the first student processing result and the second annotation information, and the initial teacher processing model and the second initial student processing model are synchronously trained based on the first total loss, wherein the first total loss is obtained based on the first prediction loss and the second prediction loss.

[0010] Exemplarily, the first initial student processing model is distilled and trained using a trained teacher processing model to obtain a trained student processing model, including: obtaining a third sample image; inputting the third sample image into the teacher encoding module of the trained teacher processing model for encoding to obtain a second teacher encoding feature; obtaining a second initial teacher query feature, the second initial teacher query feature including at least one feature vector corresponding to at least one potential target object; merging the second initial teacher query feature with the second real query feature to obtain a second teacher query feature, wherein the second real query feature is obtained by position encoding based on third annotation information, and the third annotation information is used to indicate the three-dimensional position of the target object contained in the third sample image; inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model for decoding to obtain a second teacher processing result of the third sample image, the second teacher processing result including a target detection result and / or an instance segmentation result, wherein the teacher decoding module includes a decoder module and a processing head, inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model for decoding to obtain the second teacher processing result of the third sample image, including: inputting the second teacher encoding feature and the second teacher query feature into the decoder module of the trained teacher processing model The encoder module obtains a second teacher decoding feature; inputs the second teacher decoding feature into the processing head of the trained teacher processing model to obtain a second teacher processing result; inputs the third sample image into the student encoding module of the first initial student processing model for encoding to obtain a second student encoding feature; obtains a second student query feature; inputs the second student encoding feature and the second student query feature into the student decoding module of the first initial student processing model for decoding to obtain a second student processing result of the third sample image, the second student processing result including a target detection result and / or an instance segmentation result, wherein the student decoding module includes a decoder module and a processing head, inputs the second student encoding feature and the second student query feature into the student decoding module of the first initial student processing model for decoding to obtain a second student processing result of the third sample image, including: inputting the second student encoding feature and the second student query feature into the decoder module of the first initial student processing model to obtain a second student decoding feature; inputting the second student decoding feature into the processing head of the first initial student processing model to obtain a second student processing result; determining a third prediction loss based at least on the second teacher decoding feature and the second student decoding feature; determining a second total loss based at least on the third prediction loss; optimizing the parameters in the first initial student processing model based on the second total loss to obtain a trained student processing model.

[0011] Exemplarily, the decoder module in the teacher decoding module and the decoder module in the student decoding module each include N attention layers, where N is an integer greater than 1; inputting the second teacher encoding feature and the second teacher query feature into the decoder module of the trained teacher processing model to obtain the second teacher decoding feature, including: for each current attention layer in the decoder module of the trained teacher processing model, inputting the input feature into the current attention layer to obtain the output feature, wherein, when the current attention layer is the frontmost attention layer among the N attention layers, the input feature includes the second teacher encoding feature and the second teacher query feature; when the current attention layer is any attention layer other than the frontmost attention layer among the N attention layers, the input feature includes the second teacher encoding feature and the output feature of the previous attention layer, and the output feature of the last attention layer is the second teacher decoding feature; inputting the second student encoding feature and the second student query feature into the decoder module of the first initial student processing model to obtain the second student decoding feature, including: For each current attention layer in the decoder module of the first initial student processing model, input features are input into the current attention layer to obtain output features, wherein, when the current attention layer is the frontmost attention layer among the N attention layers, the input features include the second student encoding features and the second student query features, and when the current attention layer is any attention layer other than the frontmost attention layer among the N attention layers, the input features include the second student encoding features and the output features of the previous attention layer, and the output features of the last attention layer are the second student decoding features; based at least on the second teacher decoding features and the second student decoding features, determining the third prediction loss, including: calculating the i-th sub-prediction loss based on the output features of the i-th attention layer contained in the decoder module of the trained teacher processing model and the output features of the i-th attention layer contained in the decoder module of the first initial student processing model, wherein i=1,2,3,…,N; summing or averaging the calculated N sub-prediction losses to obtain the third prediction loss.

[0012] According to another aspect of the present application, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned image processing method when the processor is executed.

[0013] According to another aspect of the present application, a storage medium is provided, on which program instructions are stored, wherein the program instructions are used to execute the above-mentioned image processing method when running.

[0014] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program, wherein the computer program is used to execute the above-mentioned image processing method when running.

[0015] According to the image processing method, electronic device, storage medium and computer program product of the embodiments of the present application, a trained student processing model is used for image processing. The student processing model is obtained by distillation training based on the teacher processing model. When training the teacher processing model, the first initial teacher query feature can be merged with the first real query feature to obtain the first teacher query feature. In this way, the teacher processing model can be trained without the need for additional multimodal annotation data such as three-dimensional point clouds, and the required training data can be greatly reduced. In addition, the position information contained in the first real query feature is more accurate, which can improve the performance of the teacher processing model obtained by training with the first teacher query feature. Using the trained teacher processing model to perform distillation training on the first initial student processing model helps to improve the feature learning ability and performance of the trained student processing model, so that the trained student processing model has better generalization and practicality. Therefore, using the student processing model trained in this way for image processing helps to improve the accuracy of the processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 A schematic block diagram illustrating an example electronic device for implementing the image processing method and apparatus according to an embodiment of the present application;

[0018] Figure 2 A schematic flowchart showing an image processing method according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram showing a flow chart of a training operation according to an embodiment of the present application is shown;

[0020] Figure 4 A diagram showing a training diagram of a teacher processing model according to one embodiment of the present application is shown;

[0021] Figure 5 A schematic diagram illustrating synchronous training of an initial teacher processing model and a second initial student processing model according to one embodiment of the present application;

[0022] Figure 6 A schematic diagram showing N attention layers according to one embodiment of the present application;

[0023] Figure 7 A schematic block diagram showing an image processing apparatus according to an embodiment of the present application; and

[0024] Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, image processing, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0026] In order to make the purpose, technical solutions and advantages of the present application more apparent, the following is a detailed description of example embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0027] The embodiments of the present application provide an image processing method, an electronic device, a storage medium and a computer program product. According to the image processing method of the embodiments of the present application, a trained student processing model is used for image processing. The student processing model is obtained by distillation training based on the teacher processing model. During training, the teacher processing model can add real query features to form teacher query features, so the student processing model can be trained and the performance of the student processing model can be guaranteed without additional data (such as three-dimensional point cloud data, etc.). The image processing technology according to the embodiments of the present application can be applied to any field involving three-dimensional target detection and / or three-dimensional instance segmentation, including but not limited to the field of autonomous driving, the field of SLAM, etc.

[0028] First, refer to Figure 1 An example electronic device 100 for implementing the image processing method and apparatus according to an embodiment of the present application is described below.

[0029] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may further include an input device 106, an output device 108, and an image acquisition device 110, and these components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as needed.

[0030] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), or a microprocessor. The processor 102 can be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or one or more combinations of other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0031] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions and / or other desired functions in the embodiments of the present application (implemented by the processor) described below. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application.

[0032] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0033] The output device 108 can output various information (e.g., images and / or sounds) to the outside (e.g., a user) and can include one or more of a display, a speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (e.g., a touch screen).

[0034] The image acquisition device 110 can capture images and store them in the storage device 104 for use by other components. The image acquisition device 110 can be a separate camera or a camera in a mobile terminal. It should be understood that the image acquisition device 110 is merely an example, and the electronic device 100 may not include the image acquisition device 110. In this case, other devices with image acquisition capabilities can be used to capture images and transmit the captured images to the electronic device 100.

[0035] For example, an example electronic device for implementing the image processing method and apparatus according to the embodiments of the present application can be implemented on a device such as a personal computer, a terminal device, an attendance machine, a panel machine, a camera, or a remote server. The terminal device includes, but is not limited to, a tablet computer, a mobile phone, a PDA (Personal Digital Assistant), an all-in-one touch-screen device, a wearable device, and the like.

[0036] Below, we will refer to Figure 2 An image processing method according to an embodiment of the present application is described. Figure 2 FIG. 2 is a schematic flow chart of an image processing method 200 according to an embodiment of the present application. Figure 2As shown, the image processing method 200 includes the following steps S210 and S220.

[0037] Step S210: obtaining an image to be processed.

[0038] The image to be processed may be an image containing any target object, such as a landscape photo, a person image or a road image. The target object may be any object, including but not limited to vehicles, pedestrians, animals, buildings, etc. In one or some embodiments of the present application, the image to be processed may be a road image, which may be a road image acquired by an image acquisition device provided on an object such as a moving vehicle, a road or a building. The image to be processed may be an original image acquired by an image acquisition device (such as the above-mentioned image acquisition device 110), or an image obtained after preprocessing the original image acquired by the image acquisition device. Preprocessing may include normalization, scaling, smoothing and other processing. Preprocessing may also include the operation of extracting a partial image area containing the target object from the original image acquired by the image acquisition device to obtain the image to be processed.

[0039] The image to be processed may be one or more images, which may be from an external device and transmitted by the external device to the electronic device 100 for image processing. Furthermore, the image to be processed may also be acquired by the electronic device 100 itself. For example, the electronic device 100 may utilize an image acquisition device 110 (e.g., an independent camera) to acquire the image to be processed. The image acquisition device 110 may transmit the acquired image to be processed to the processor 102, which then performs image processing.

[0040] In step S220 , the trained student processing model is used to perform image processing on the image to be processed to obtain a processing result of the image to be processed, where the image processing includes three-dimensional target detection and / or three-dimensional instance segmentation, and the processing result includes a target detection result and / or an instance segmentation result.

[0041] Exemplarily, the obtained image to be processed is input into a trained student processing model to obtain the corresponding processing result. Image processing can include any one or more of 3D object detection and 3D instance segmentation. For 3D object detection, the object detection result can exemplarily include the position information of the 3D detection box of the target object (i.e., the target detection box) and the confidence level corresponding to each 3D detection box. The 3D detection box is a bounding box that contains the target object and can optionally be a rectangular box. Of course, the shape of the target detection box can also be other suitable shapes, such as a sphere or a cone. The same target object can correspond to one or more target detection boxes. The position information of the detection box can include one or more of the following information of the detection box: corner coordinates of one or more corner points; center coordinates; length information; width information; and height information. Where the position information of the detection box includes one or more of length information, width information, and height information, the position information can further include corner coordinates and / or center coordinates of at least one corner point. The confidence level of the detection box can be represented by any numerical value, for example, the confidence level can range from 0 to 1. As mentioned above, the closer the confidence value is to 1, the more accurate the target object detected by the detection box is. Similarly, the final position information of the detection box and its confidence can be understood by reference. For three-dimensional instance segmentation, illustratively, the instance segmentation result may include mask information of the target object. The mask information is used to indicate the position of the corresponding three-dimensional mask of the target object. The mask information can be presented through a heat map. In the heat map, each pixel within the mask of the target object can be highlighted.

[0042] The trained student processing model is obtained through training operation. Figure 3 FIG. 5 is a flow chart showing a training operation S300 according to an embodiment of the present application. Figure 3 , the training operation S300 includes steps S310, S320, S330, S340, S350, S360 and S370.

[0043] Step S310 : Acquire a first sample image and corresponding first annotation information, where the first annotation information is used to indicate a three-dimensional position of a target object contained in the first sample image.

[0044] The first sample image and the first annotation information are used to train the teacher processing model. Figure 4A training diagram of a teacher processing model according to an embodiment of the present application is shown. Exemplarily, the method for acquiring the first sample image X1 is similar to the method for acquiring the image to be processed. The method for acquiring the image to be processed has been described in detail in step S210, and for the sake of brevity, it will not be repeated here. For the application scenario of three-dimensional target detection, the first annotation (ground truth) information may include the position information of the real detection frame, which is used to indicate the three-dimensional position of the target object contained in the first sample image. For example, the three-dimensional position of the target object contained in the first sample image can be annotated by a rectangular detection frame. For the application scenario of three-dimensional instance segmentation, the first annotation information may include mask information of the real mask of the target object contained in the first sample image. For the sake of ease of description and understanding, the application scenario of the image processing method is mainly described below as three-dimensional target detection. The implementation method of three-dimensional instance segmentation is similar to that of three-dimensional target detection.

[0045] Step S320: Input the first sample image into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature.

[0046] In one embodiment, the first sample image X1 is input into the teacher encoding module of the initial teacher processing model for encoding, and the first teacher encoding feature M can be obtained. T1 The initial teacher processing model can be any 3D object detection or 3D instance segmentation model, such as a Position Embedding Transformation (PETR) model, a Detection Transformer (DETR) model, etc. The teacher encoding module can include the encoder in the PETR or DETR model.

[0047] Step S330: Obtain a first initial teacher query feature, where the first initial teacher query feature includes at least one feature vector corresponding one-to-one to at least one potential target object.

[0048] The first initial teacher query feature Q1 can be a predefined feature sequence. The first initial teacher query feature can be expressed as The first initial teacher query feature Q1 can include the original learned feature vectors in the PETR framework.

[0049] Step S340: Merge the first initial teacher query feature and the first real query feature to obtain the first teacher query feature, wherein the first real query feature is obtained by position encoding based on the first annotation information.

[0050] Based on the first annotation information contained in the first sample image, the position of each annotated real target object can be encoded to obtain the corresponding first real query feature Q2. The first real query feature can be expressed as It is composed of L2 learnable feature vectors of length C, and the L2 learnable feature vectors of length C are feature vectors corresponding one to one to L2 real target objects. L2 and L1 can be equal or unequal. For example, the initial teacher processing model can be a three-dimensional target detection model, the first sample image can be a multi-view image, and the first annotation information can include the position information of the three-dimensional target detection box corresponding to the target object under the bird's eye view (BEV). The three-dimensional position coordinates of the target object can be obtained by a position encoding network (for example, using triangular periodic function mapping plus fully connected layer mapping) to obtain the first real query feature Q2. The first initial teacher query feature Q1 and the first real query feature Q2 are merged to obtain the first teacher query feature Q T1 The first teacher query feature Q T1 ∈R L×C , consisting of L learnable feature vectors of length C, L = L1 + L2. The first teacher query feature Q T1 , the number of channels C of the first initial teacher query feature Q1 and the first real query feature Q2 are equal.

[0051] Step S350, input the first teacher encoding feature and the first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding, and obtain the first teacher processing result of the first sample image, the first teacher processing result including the target detection result and / or the instance segmentation result.

[0052] Refer again Figure 4 , for the teacher decoding module in the initial teacher processing model, the first teacher encoding feature M T1 and the first teacher query feature Q T1 As the input of the teacher decoding module, the first teacher processing result is obtained. For example, the first teacher processing result may include the position information of the predicted target detection box of the target object in the first sample image or the mask information of the predicted mask. For example, the teacher decoding module may include the decoder in the PETR or DETR model.

[0053] Step S360: Training the initial teacher processing model based at least on the first teacher processing result and the first annotation information to obtain a trained teacher processing model.

[0054] For example, based on the obtained first teacher processing results and the first annotation information, the prediction loss of the initial processing model can be calculated, and the parameters of the initial teacher detection model can be optimized through backpropagation and gradient descent algorithms to obtain a trained teacher processing model. For example, during the process of optimizing the parameters of the initial teacher processing model, the first initial teacher query feature can also be optimized simultaneously.

[0055] Step S370: Use the trained teacher processing model to perform distillation training on the first initial student processing model to obtain a trained student processing model.

[0056] Step S370 can employ any existing or future distillation training method to perform distillation training on the first initial student processing model. The trained first initial student processing model is referred to as the trained student processing model. During distillation, the trained teacher processing model can be loaded and its parameters fixed to perform distillation training on the first initial student processing model.

[0057] In one or some embodiments, the image processing method 200 can be applied to the field of autonomous driving, and the student processing model and the teacher processing model can optionally be 3D object detection models. In this case, the number of images to be processed can be multiple, and the multiple images to be processed can be multi-angle images collected from different angles.

[0058] According to the image processing method of the embodiment of the present application, a trained student processing model is used for image processing. The student processing model is obtained by distillation training based on the teacher processing model. When training the teacher processing model, the first initial teacher query feature and the first real query feature can be merged to obtain the first teacher query feature. In this way, the teacher processing model can be trained without the need for additional multimodal annotation data such as three-dimensional point clouds, and the required training data can be greatly reduced. In addition, the position information contained in the first real query feature is more accurate, which can improve the performance of the teacher processing model obtained by training with the first teacher query feature. Using the trained teacher processing model to perform distillation training on the first initial student processing model helps to improve the feature learning ability and performance of the trained student processing model, so that the trained student processing model has better generalization and practicality. Therefore, using the student processing model trained in this way for image processing helps to improve the accuracy of the processing results.

[0059] Exemplarily, the image processing method according to the embodiments of the present application may be implemented in a device, apparatus, or system having a memory and a processor.

[0060] The image processing method according to the embodiment of the present application can be deployed at an image acquisition end, for example, can be deployed at a personal terminal or a server end with an image acquisition function.

[0061] Alternatively, the image processing method according to the embodiments of the present application can also be deployed in a distributed manner on the server (or cloud) and the personal terminal. For example, an image can be acquired on the client, and the client transmits the acquired image to the server (or cloud), which then processes the image.

[0062] Exemplarily, before using the trained teacher processing model to perform distillation training on the first initial student processing model to obtain the trained student processing model, the training operation may also include: obtaining a second sample image and corresponding second annotation information, the second annotation information being used to indicate the three-dimensional position of the target object contained in the second sample image; inputting the second sample image into the student encoding module of the second initial student processing model for encoding to obtain a first student encoding feature; obtaining a first student query feature, the first student query feature including at least one feature vector corresponding one-to-one to at least one potential target object; inputting the first student encoding feature and the first student query feature into the student decoding module of the second initial student processing model for decoding to obtain a first student processing result of the second sample image, the first student processing result including a target detection result and / or an instance segmentation result; training the second initial student processing model based at least on the first student processing result and the second annotation information to obtain the first initial student processing model.

[0063] The first student query feature can be a predefined feature sequence. The first student query feature can be expressed as It is composed of L3 learnable feature vectors of length C, and the L3 learnable feature vectors of length C are feature vectors corresponding one-to-one to L3 potential target objects.

[0064] The second sample image and the first sample image can be the same image or different images. In one embodiment, before using the trained teacher processing model to perform distillation training on the first initial student processing model to obtain the trained student processing model, the training operation can also include training the second initial student processing model to obtain the first initial student processing model. A person skilled in the art can understand the training method of the initial student processing model by reading the relevant description of the initial teacher processing model in the previous embodiment. For the sake of brevity, it is not repeated here. In the process of training the second initial student processing model based on the first student processing result and the second annotation information, it can also include training the obtained first student query features to optimize the first student query features. The network structure of the second initial student processing model and the first initial student processing model can be consistent, but the size of the parameters (including weights and / or biases, etc.) in the two may not be completely consistent. Similarly, the network structure of the first initial student processing model and the trained student processing model can be consistent, but the size of the parameters in the two may not be completely consistent. The training process of the initial teacher processing model and the training process of the second initial student processing model can be performed simultaneously or independently. In addition, the operation of obtaining the first initial student processing model through training the second initial student processing model is optional. The first initial student processing model can be a preset student processing model, that is, the parameters in the first initial student processing model can be preset.

[0065] According to the above technical solution, the first student processing result is obtained through the second sample image and the corresponding second annotation information, and then the second initial student processing model is trained based on the first student processing result and the second annotation information to obtain the first initial student processing model. This method can further ensure the performance of the first initial student processing model and its feature learning ability.

[0066] Exemplarily, the teacher encoding module may include a feature extraction module and a position encoder module. Inputting a first sample image into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature may include: inputting the first sample image into the feature extraction module of the initial teacher processing model for feature extraction to obtain a first teacher image feature; obtaining a first position embedded feature corresponding to the first sample image; inputting the first teacher image feature and the first position embedded feature into the position encoder module of the initial teacher processing model for position encoding to obtain the first teacher encoding feature. Exemplarily, the student encoding module may include a feature extraction module and a position encoder module. Inputting a second sample image into the student encoding module of the second initial student processing model for encoding to obtain a first student encoding feature may include: inputting the second sample image into the feature extraction module of the second initial student processing model for feature extraction to obtain a first student image feature; obtaining a second position embedded feature corresponding to the second sample image; inputting the first student image feature and the second position embedded feature into the position encoder module of the second initial student processing model for position encoding to obtain a first student encoding feature. Wherein, the feature extraction module of the initial teacher processing model and the feature extraction module of the second initial student processing model are the same shared feature extraction module.

[0067] In one embodiment, the first sample image X1 can be expressed as Where H0, W0, and C0 represent the height, width, and number of channels of the first sample image X1, respectively. For example, when the first sample image X1 is an RGB image, the number of channels C0 can be 3. The first sample image X1 can be input into the feature extraction module of the initial teacher processing model, and the feature extraction module can extract features of the first sample image X1 to obtain the first teacher image feature F T , F T ∈R H×W×C Where H, W and C represent the first teacher image features F T The height, width and number of channels of the first sample image can be obtained by illustratively but not restrictively implementing the feature extraction module using a convolutional neural network backbone (CNN backbone). T . The first teacher image feature F T and the first position embedding feature P T Input the position encoder module of the initial teacher processing model together for position encoding, and the first teacher encoding feature M can be obtained. T1 .

[0068] Similarly, the first student image feature F can be obtained in a similar manner to the above embodiment. S And the second position embedding feature P corresponding to the second sample image S . The first student image feature F S and the second position embedding feature P S Input the position encoder module of the second initial student processing model for position encoding, and the first student encoding feature M can be obtained. S1 . The feature extraction module of the initial teacher processing model and the feature extraction module of the second initial student processing model can be the same shared feature extraction module. That is, the initial teacher processing model and the second initial student processing model can share a feature extraction module. It can be understood that since the network structure of the initial teacher processing model is consistent with that of the trained teacher processing model, and the network structure of the first initial student processing model and the second initial student processing model is consistent, the trained teacher processing model and the first initial student processing model can also share the same feature extraction module.

[0069] According to the above technical solution, the feature extraction module of the initial teacher processing model and the feature extraction module of the second initial student processing model are the same shared feature extraction module. The image processing model obtained thereby is a lightweight model, which can effectively reduce the amount of parameters, reduce the data training cost, and improve the model training and processing efficiency.

[0070] Exemplarily, the teacher decoding module includes a decoder module and a processing head, the processing head of the teacher decoding module includes a detection head for outputting target detection results and / or a segmentation head for outputting instance segmentation results, and the first teacher encoding feature and the first teacher query feature are input into the teacher decoding module of the initial teacher processing model for decoding to obtain the first teacher processing result of the first sample image, which may include: inputting the first teacher encoding feature and the first teacher query feature into the decoder module of the initial teacher processing model to obtain the first teacher decoding feature; inputting the first teacher decoding feature into the processing head of the initial teacher processing model to obtain the first teacher processing result. Exemplarily, the student decoding module includes a decoder module and a processing head, the processing head of the student decoding module includes a detection head for outputting target detection results and / or a segmentation head for outputting instance segmentation results, and the first student encoding feature and the first student query feature are input into the student decoding module of the second initial student processing model for decoding to obtain the first student processing result of the second sample image, which may include: inputting the first student encoding feature and the first student query feature into the decoder module of the second initial student processing model to obtain the first student decoding feature; inputting the first student decoding feature into the processing head of the second initial student processing model to obtain the first student processing result; wherein, the processing head of the initial teacher processing model and the processing head of the second initial student processing model are the same shared processing head.

[0071] In one embodiment, the teacher decoding module may include a decoder module and a processing head, wherein the processing head includes a detection head and / or a segmentation head. For applications of the image processing method in a three-dimensional object detection scenario, the processing head is a detection head. For applications of the image processing method of the present embodiment in a three-dimensional instance segmentation scenario, the processing head is a segmentation head. In addition, the processing head may also include both a detection head and a segmentation head.

[0072] The following is an example of the application of the image processing method of the embodiment of the present application in the three-dimensional target detection scene. T1 and the first teacher query feature Q T1 Input the decoder module of the initial teacher processing model to obtain the first teacher decoding feature Q T . Exemplarily, the decoder module of the initial teacher processing model can be implemented using the decoder module in the PETR framework.

[0073] Similarly, referring to the above embodiment, one can understand the implementation method of obtaining the first student processing result of the second sample image. For the sake of brevity, it will not be repeated here. The processing head of the initial teacher processing model and the processing head of the second initial student processing model can be the same shared processing head. That is, the initial teacher processing model and the second initial student processing model can share one processing head. It can be understood that since the network structure of the initial teacher processing model is consistent with that of the trained teacher processing model, and the network structure of the first initial student processing model and the second initial student processing model is consistent, the trained teacher processing model and the first initial student processing model can also share the same processing head.

[0074] According to the above technical solution, the processing head of the initial teacher processing model and the processing head of the second initial student processing model are the same shared processing head. The image processing model obtained is a lightweight model, which can effectively reduce the number of parameters, reduce the data training cost, and improve the model training and processing efficiency.

[0075] Exemplarily, the second sample image is the same as the first sample image, wherein, in the process of training the initial teacher processing model based at least on the first teacher processing result and the first annotation information, the first prediction loss is determined based on the first teacher processing result and the first annotation information, and in the process of training the second initial student processing model based at least on the first student processing result and the second annotation information, the second prediction loss is determined based on the first student processing result and the second annotation information, and the initial teacher processing model and the second initial student processing model are synchronously trained based on the first total loss, wherein the first total loss is obtained based on the first prediction loss and the second prediction loss.

[0076] Figure 5FIG. 1 shows a schematic diagram of synchronous training of an initial teacher processing model and a second initial student processing model according to an embodiment of the present application. Figure 5 As shown, the first sample image is input into the shared feature extraction module to obtain the first teacher image feature F T and the first student image feature F S In addition, the first position embedding feature P can be obtained T and the second position embedding feature P S . The first teacher image feature F T and the first position embedding feature P T Input the teacher coding module of the initial teacher processing model to obtain the corresponding first teacher coding feature M T1 . Using the first teacher encoding feature M T1 and the first teacher query feature Q T1 Input into the teacher decoding module, the corresponding first teacher decoding feature Q can be obtained T . The first teacher decoding feature Q T Input to the shared processing head to obtain the first teacher processing result. Similarly, the first student processing result can be obtained by using the second initial student processing model in a similar way. S and the second position embedding feature P S Input the student coding module of the second initial student processing model to obtain the corresponding first student coding feature M S1 . Using the first student encoding feature M S1 and the first student query feature Q S1 Input into the student decoding module, the corresponding first student decoding feature Q can be obtained S . The first student decoding feature Q S Input to the shared processing header to obtain the first student processing results.

[0077] In one embodiment, the first sample image and the second sample image are the same. The training of the initial teacher processing model and the training of the second initial student processing model can be performed simultaneously. For the training of the initial teacher processing model, the first prediction loss LS1 between the two can be calculated based on the first teacher processing result and the first annotation information. For the training of the second initial student processing model, the second prediction loss LS2 between the two can be calculated based on the first student processing result and the second annotation information. The first total loss L can be obtained by summing or averaging the first prediction loss LS1 and the second prediction loss LS2. 1 Based on the first total loss L 1, the back propagation and gradient descent algorithms can be used to simultaneously optimize the parameters in the initial teacher processing model and the second initial student processing model, thereby obtaining the trained teacher processing model and the first initial student processing model.

[0078] According to the above technical solution, the initial teacher processing model and the second initial student processing model are trained synchronously based on the first total loss obtained based on the first prediction loss and the second prediction loss. This method can avoid overfitting of the shared feature extraction network or processing head to the teacher processing model.

[0079] Exemplarily, training the initial teacher processing model based at least on the first teacher processing result and the first annotation information to obtain a trained teacher processing model may include: synchronously optimizing the parameters in the initial teacher processing model and the first initial teacher query feature based at least on the first teacher processing result and the first annotation information to obtain a trained teacher processing model and a trained initial teacher query feature.

[0080] In one embodiment, based on the first prediction loss calculated based on the first teacher processing result and the first annotation information, or based on the first total loss, in addition to optimizing the parameters in the initial teacher processing model, the first initial teacher query features can also be optimized synchronously. This can improve the convergence efficiency of the initial teacher processing model.

[0081] Exemplarily, using a trained teacher processing model to perform distillation training on a first initial student processing model to obtain a trained student processing model may include: obtaining a third sample image; inputting the third sample image into the teacher encoding module of the trained teacher processing model for encoding to obtain a second teacher encoding feature; obtaining a second initial teacher query feature, the second initial teacher query feature including at least one feature vector corresponding to at least one potential target object; merging the second initial teacher query feature with the second real query feature to obtain a second teacher query feature, wherein the second real query feature is obtained by position encoding based on third annotation information, and the third annotation information is used to indicate the three-dimensional position of the target object contained in the third sample image; inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model for decoding to obtain a second teacher processing result of the third sample image, the second teacher processing result including a target detection result and / or an instance segmentation result, wherein the teacher decoding module includes a decoder module and a processing head, inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model for decoding to obtain the second teacher processing result of the third sample image, which may include: inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model A decoder module obtains a second teacher decoding feature; inputs the second teacher decoding feature into the processing head of the trained teacher processing model to obtain a second teacher processing result; inputs the third sample image into the student encoding module of the first initial student processing model for encoding to obtain a second student encoding feature; obtains a second student query feature; inputs the second student encoding feature and the second student query feature into the student decoding module of the first initial student processing model for decoding to obtain a second student processing result of the third sample image, the second student processing result including a target detection result and / or an instance segmentation result, wherein the student decoding module includes a decoder module and a processing head, inputs the second student encoding feature and the second student query feature into the student decoding module of the first initial student processing model for decoding to obtain a second student processing result of the third sample image, which may include: inputting the second student encoding feature and the second student query feature into the decoder module of the first initial student processing model to obtain a second student decoding feature; inputting the second student decoding feature into the processing head of the first initial student processing model to obtain a second student processing result; determining a third prediction loss based at least on the second teacher decoding feature and the second student decoding feature; determining a second total loss based at least on the third prediction loss; optimizing the parameters in the first initial student processing model based on the second total loss to obtain a trained student processing model.

[0082] In one embodiment, the method for obtaining the third sample image is similar to the method for obtaining the first sample image. The method for obtaining the first sample image has been described in detail in step S210 and will not be repeated here for the sake of brevity. A person of ordinary skill in the art can understand the implementation method for obtaining the second teacher processing result and the second student processing result through the third sample image by reading the relevant descriptions in the previous embodiments. Among them, the second initial teacher query feature can be the first initial teacher query feature or the trained initial teacher query feature (that is, the result obtained after training the first initial teacher query feature). The third sample image can be the same as or different from the above-mentioned first sample image or second sample image.

[0083] The second initial teacher query feature Q3 can be a predefined feature sequence. The second initial teacher query feature can be expressed as It is composed of L4 learnable feature vectors of length C, and the L4 learnable feature vectors of length C are feature vectors corresponding one to one to L4 potential target objects. Wherein, C represents the number of channels. The second real query feature is obtained by position encoding based on the third annotation information corresponding to the third sample image, and its acquisition method is similar to the acquisition method of the first real query feature, which will not be repeated here. The method of obtaining the second teacher query feature based on the second initial teacher query feature and the second real query feature is similar to the method of obtaining the first teacher query feature based on the first initial teacher query feature and the first real query feature, which will not be repeated here. In addition, the method of obtaining the second teacher processing result, the second student coding feature, the second student query feature, the second student processing result and other information can be understood by referring to the method of obtaining the first teacher processing result, the first student coding feature, the first student query feature, the first student processing result and other information described above, which will not be repeated here.

[0084] Based on the obtained second teacher decoding features and second student decoding features, a third prediction loss can be calculated. Exemplarily and non-restrictively, the third prediction loss can be a mean square error (MSE) loss function, i.e., an L2 loss function. The second total loss can be determined based on at least the third prediction loss. For example, the third prediction loss can be directly determined as the second total loss, and the second total loss can be further determined in combination with other losses. Using the second total loss, the parameters in the first initial student processing model are optimized through back propagation and gradient descent algorithms, and then a trained student processing model can be obtained.

[0085] According to the above technical solution, the parameters in the first initial student processing model are optimized based on the second total loss to obtain a trained student processing model. The trained student processing model obtained by this method can have better feature learning ability and performance.

[0086] Exemplarily, the decoder module in the teacher decoding module and the decoder module in the student decoding module each include N attention layers, where N is an integer greater than 1; inputting the second teacher encoding feature and the second teacher query feature into the decoder module of the trained teacher processing model to obtain the second teacher decoding feature may include: for each current attention layer in the decoder module of the trained teacher processing model, inputting the input feature into the current attention layer to obtain the output feature, wherein, when the current attention layer is the frontmost attention layer among the N attention layers, the input feature includes the second teacher encoding feature and the second teacher query feature; when the current attention layer is any attention layer among the N attention layers except the frontmost attention layer, the input feature includes the second teacher encoding feature and the output feature of the previous attention layer, and the output feature of the last attention layer is the second teacher decoding feature; exemplarily, inputting the second student encoding feature and the second student query feature into the decoder module of the first initial student processing model to obtain the second student decoding feature may include : For each current attention layer in the decoder module of the first initial student processing model, input features are input into the current attention layer to obtain output features, wherein, when the current attention layer is the frontmost attention layer among the N attention layers, the input features include the second student encoding features and the second student query features, and when the current attention layer is any attention layer other than the frontmost attention layer among the N attention layers, the input features include the second student encoding features and the output features of the previous attention layer, and the output features of the last attention layer are the second student decoding features; exemplarily, determining the third prediction loss based on at least the second teacher decoding features and the second student decoding features may include: calculating the i-th sub-prediction loss based on the output features of the i-th attention layer contained in the decoder module of the trained teacher processing model and the output features of the i-th attention layer contained in the decoder module of the first initial student processing model, wherein i=1,2,3,…,N; summing or averaging the calculated N sub-prediction losses to obtain the third prediction loss.

[0087] In one embodiment, the decoder module in the teacher decoding module and the decoder module in the student decoding module may both include N attention layers, for example, N may be equal to 6. The following description will be made by taking the decoder module in the teacher decoding module as an example.

[0088] Figure 6 FIG. 1 shows a schematic diagram of N attention layers according to an embodiment of the present application. Figure 6 As shown, for the attention layer at the front of the N attention layers (i.e., the first attention layer), its corresponding input feature is the second teacher encoding feature M T2 and the second teacher query feature Q T2. The second teacher encoding feature M T2 and the second teacher query feature Q T2 Input the first attention layer to obtain the output feature Q corresponding to the attention layer T21 The output feature Q T21 And the second teacher coding feature M T2 It can be used as the input feature of the next attention layer (i.e., the second attention layer), and then the output feature Q corresponding to the second attention layer is obtained. T22 In a similar way, the output features corresponding to the previous attention layer and the second teacher encoding features M are sequentially T As the input feature of the next attention layer, it is input to the next attention layer, and finally the output feature Q corresponding to the 6th attention layer can be obtained. T26 The output feature Q corresponding to the 6th attention layer T26 It can be used as the final decoding feature of the decoder module in the teacher decoding module, that is, the second teacher decoding feature Q T '.

[0089] Similarly, those skilled in the art can understand the implementation of the N attention layers of the decoder module in the student decoding module by understanding the implementation of the N attention layers of the decoder module in the teacher decoding module. For the sake of brevity, we will not go into details here. The output features corresponding to the 6 attention layers of the decoder module in the student decoding module are Q S21 , Q S22 ,...,Q S26 .

[0090] Based on the output features of any attention layer included in the decoder module of the trained teacher processing model and the output features of the corresponding attention layer included in the decoder module of the first initial student processing model, the sub-prediction loss of the current attention layer can be calculated. For example, for the second attention layer, based on the output feature Q of the second attention layer included in the decoder module of the trained teacher processing model T22 And the output feature Q of the second attention layer included in the decoder module of the first initial student processing model S22 , the sub-prediction loss l2 of the current second attention layer can be calculated. Similarly, the sub-prediction losses l1, l2, ..., l6 corresponding to each attention layer can be obtained in sequence. By summing or averaging the first to sixth sub-prediction losses, the third prediction loss LS3 can be obtained.

[0091] According to the above technical solution, the third prediction loss is calculated based on the sub-prediction losses corresponding to each attention layer in the decoder module of the teacher decoding module and the decoder module of the student decoding module. This allows the output features of each attention layer to be supervised by the loss from the teacher processing model to the student processing model. This can further improve the feature learning ability of the student processing model.

[0092] Exemplarily, obtaining a third sample image may include: obtaining a third sample image and corresponding third annotation information, the third annotation information being used to indicate the three-dimensional position of the target object contained in the third sample image. Exemplarily, before determining the second total loss based at least on the third prediction loss, performing distillation training on the first initial student processing model using the trained teacher processing model to obtain a trained student processing model may also include one or more of the following: determining a fourth prediction loss based on the second student processing result and the third annotation information; determining a fifth prediction loss based on the second teacher processing result and the third annotation information; and determining a sixth prediction loss based on the second student processing result and the second teacher processing result. Exemplarily, determining the second total loss based at least on the third prediction loss may include: determining the second total loss based on the third prediction loss and based on one or more of the fourth prediction loss, the fifth prediction loss, and the sixth prediction loss.

[0093] In one embodiment, the third annotation information corresponding to the third sample image can be used to indicate the 3D position of the target object contained in the third sample image. Similarly, in the application scenario of 3D object detection, the third annotation information can include the position information and confidence level of the 3D detection box corresponding to the target object; in the application scenario of 3D instance segmentation, the third annotation information can include the 3D mask information corresponding to the target object.

[0094] The fourth predicted loss LS4 can be calculated by the second student processing result and the third labeled information. The fifth predicted loss LS5 can be calculated by the second teacher processing result and the third labeled information. The sixth predicted loss LS6 can be calculated by the second student processing result and the second teacher processing result. By way of example and not limitation, the second total loss L 2 It can be obtained by summing or averaging one or more of the fourth prediction loss LS4, the fifth prediction loss LS5, and the sixth prediction loss LS6 with the third prediction loss LS3. 2 =LS3+LS4+LS6 or L 2 =LS3+LS4.

[0095] According to the above technical solution, the second total loss is determined based on one or more of the fourth prediction loss, the fifth prediction loss and the sixth prediction loss and the third prediction loss, which can further improve the performance of the trained student process.

[0096] Exemplarily, the second teacher processing result may include the target detection result, the second student processing result may include the target detection result, the second teacher decoding feature includes M1 groups of feature vectors corresponding one-to-one to M1 target detection frames, the second teacher processing result includes M1 groups of detection frame information corresponding one-to-one to M1 target detection frames, the second student decoding feature includes M2 groups of feature vectors corresponding one-to-one to M2 target detection frames, the second student processing result includes M2 groups of detection frame information corresponding one-to-one to M2 target detection frames, M1 is an integer greater than or equal to 1, and M2 is an integer greater than or equal to 1; Exemplarily, obtaining a third sample image may include: obtaining a third sample image and corresponding third annotation information, the third annotation information is used to indicate the three-dimensional position of the target object contained in the third sample image; before determining the third prediction loss based at least on the second teacher decoding feature and the second student decoding feature, using the trained teacher processing model to perform distillation training on the first initial student processing model to obtain the trained student processing model, and may also include: comparing the target detection frame in the second teacher processing result with the target detection frame in the third annotation information to obtain a first matching result, wherein the target detection frame is a bounding box containing any target object; the target detection frame in the second student processing result is matched with the target detection frame in the third annotation information to obtain a second matching result; based on the first matching result and the second matching result, a third matching result between the target detection frame in the second teacher processing result and the target detection frame in the second student processing result is determined, the third matching result including M3 pairs of mutually matching detection frame pairs, M3 is an integer greater than or equal to 0 and M3 is less than or equal to the smaller of M1 and M2, each pair of detection frame pairs including a first target detection frame belonging to the second teacher processing result and a second target detection frame belonging to the second student processing result; exemplarily, determining the third prediction loss based at least on the second teacher decoding feature and the second student decoding feature may include: determining a seventh prediction loss based on the feature vectors in the second teacher decoding feature corresponding one-to-one to the M3 first target detection frames and the feature vectors in the second student decoding feature corresponding one-to-one to the M3 second target detection frames; determining the third prediction loss based at least on the seventh prediction loss.

[0097] In one embodiment, the second teacher processing result and the second student processing result may include target detection results. The target detection results include target detection frames corresponding one-to-one to multiple target objects and the confidence level of each target detection frame. The second teacher decoding feature may include M1 sets of feature vectors corresponding one-to-one to M1 target detection frames. For example, M1 may be equal to 20. Then the second teacher processing result may include 20 sets of detection frame information corresponding one-to-one to the 20 target detection frames. The detection frame information may include the location information of the target detection frames and an optional confidence level. Similarly, the second student decoding feature may include M2 ​​sets of feature vectors corresponding one-to-one to M2 target detection frames. For example, M2 may be equal to 15. Then the second student processing result may include 15 sets of detection frame information corresponding one-to-one to the 15 target detection frames. Wherein, M1 and M2 are both integers greater than or equal to 1, and M1 and M2 may be equal or unequal, that is, M1 and M2 may both be equal to 20, or M1 may be equal to 20 and M2 may be equal to 15.

[0098] In one embodiment, before determining the third prediction loss based on at least the second teacher decoding feature and the second student decoding feature, the first initial student processing model is distilled and trained using the trained teacher processing model to obtain the trained student processing model, which may also include the following operations.

[0099] The target detection frames in the second teacher processing result are matched with the target detection frames in the third annotation information, i.e., target detection frames corresponding to the same target object are matched, thereby obtaining a first matching result. Similarly, the target detection frames in the second student processing result are matched with the target detection frames in the third annotation information to obtain a second matching result. Based on the first and second matching results obtained, target detection frames containing the same target object in both are matched again to obtain a third matching result. The number M3 of mutually matching detection frame pairs included in the third matching result is an integer greater than 0 and less than or equal to the smaller of M1 and M2. For example, M3 can be equal to 5. A seventh prediction loss is calculated based on the feature vectors in the second teacher decoded features that correspond one-to-one to the five first target detection frames and the feature vectors in the second student decoded features that correspond one-to-one to the five second target detection frames. In the above-mentioned embodiment using multiple attention layers, the seventh prediction loss can be the sub-prediction loss corresponding to the last attention layer of the N attention layers included in the decoder module in the trained teacher processing model and the decoder module in the first initial student processing model, such as the sixth sub-prediction loss l6 in the above-mentioned embodiment. The current sub-prediction loss l6 may be determined as the third prediction loss, or multiple sub-prediction losses may be summed and the summed result may be determined as the third prediction loss, for example, LS3 = l1 + l2 + l3 + l4 + l5 + l6.

[0100] According to the above technical solution, based on the first matching result and the second matching result, a third matching result between the target detection frame in the second teacher processing result and the target detection frame in the second student processing result is determined. In this way, when the first initial student processing model is distilled and trained, loss supervision can be mainly performed based on the information of the foreground (i.e., mutually matching) target detection frames, while excluding the interference of the background (non-matching) target detection frame information. This can improve the accuracy of the obtained trained student processing model.

[0101] Research has shown that this training operation can improve the performance ceiling of existing 3D object detection frameworks. Experimental results on the NuScenes 3D object detection task show that, compared to baseline methods, the student processing model obtained through this application's training method can improve the maximum a posteriori probability (MAP) by 0.7% and the NuScenes Detection Score (NDS) by 2.0%. It is expected to achieve comparable performance gains on real-world business datasets.

[0102] According to another aspect of the present application, an image processing device is provided. Figure 7 FIG2 shows a schematic block diagram of an image processing apparatus 700 according to an embodiment of the present application.

[0103] like Figure 7 As shown, the image processing device 700 according to the embodiment of the present application includes an acquisition module 710 and a processing module 720. Each module can respectively perform the above Figure 2 The following only describes the main functions of the components of the image processing device 700, and omits the details already described above.

[0104] The acquisition module 710 is used to acquire the image to be processed. The acquisition module 710 can be Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement the above.

[0105] The processing module 720 is used to perform image processing on the image to be processed using the trained student processing model to obtain a processing result of the image to be processed. The image processing includes three-dimensional target detection and / or three-dimensional instance segmentation, and the processing result includes target detection result and / or instance segmentation result. The trained student processing model is obtained through the following training operations: obtaining a first sample image and the corresponding first annotation information, the first annotation information is used to indicate the three-dimensional position of the target object contained in the first sample image; inputting the first sample image into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature; obtaining a first initial teacher query feature, the first initial teacher query feature includes at least one feature vector corresponding to at least one potential target object; merging the first initial teacher query feature with the first real query feature to obtain a first teacher query feature, wherein the first real query feature is obtained by position encoding based on the first annotation information; inputting the first teacher encoding feature and the first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding to obtain a first teacher processing result of the first sample image, the first teacher processing result includes a target detection result and / or an instance segmentation result; training the initial teacher processing model based on at least the first teacher processing result and the first annotation information to obtain a trained teacher processing model; using the trained teacher processing model to perform distillation training on the first initial student processing model to obtain a trained student processing model. The processing module 720 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement the above.

[0106] Figure 8 FIG2 is a schematic block diagram of an electronic device 800 according to an embodiment of the present application. The electronic device 800 includes a memory 810 and a processor 820 .

[0107] The memory 810 stores computer program instructions for implementing corresponding steps in the image processing method according to the embodiment of the present application.

[0108] The processor 820 is configured to execute computer program instructions stored in the memory 810 to perform corresponding steps of the image processing method according to the embodiment of the present application.

[0109] In one embodiment, the computer program instructions are used by the processor 820 to execute the following steps when the processor 820 is running: obtaining an image to be processed; performing image processing on the image to be processed using the trained student processing model to obtain a processing result of the image to be processed, wherein the image processing includes three-dimensional target detection and / or three-dimensional instance segmentation, and the processing result includes a target detection result and / or an instance segmentation result; the trained student processing model is obtained by training the following training operations: obtaining a first sample image and corresponding first annotation information, wherein the first annotation information is used to indicate the three-dimensional position of the target object contained in the first sample image; inputting the first sample image into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature; obtaining a first initial teacher query feature, wherein the first initial teacher query feature includes At least one feature vector corresponding one-to-one to at least one potential target object; merging a first initial teacher query feature with a first real query feature to obtain a first teacher query feature, wherein the first real query feature is obtained by position encoding based on the first annotation information; inputting the first teacher encoding feature and the first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding to obtain a first teacher processing result of the first sample image, the first teacher processing result including a target detection result and / or an instance segmentation result; training the initial teacher processing model based on at least the first teacher processing result and the first annotation information to obtain a trained teacher processing model; and performing distillation training on the first initial student processing model using the trained teacher processing model to obtain a trained student processing model.

[0110] For example, the electronic device 800 may further include an image acquisition device 830. The image acquisition device 830 is used to acquire images to be processed. The image acquisition device 830 is optional, and the electronic device 800 may not include the image acquisition device 830. In this case, the processor 820 may obtain the images to be processed by other means, such as from an external device or from the memory 810.

[0111] In addition, according to an embodiment of the present application, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the program instructions are used to execute the corresponding steps of the image processing method according to the embodiment of the present application, and are used to implement the corresponding modules in the image processing device according to the embodiment of the present application. The storage medium may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0112] In one embodiment, when the program instructions are executed by a computer or processor, the computer or processor may implement the various functional modules of the image processing device according to the embodiment of the present application, and / or may execute the image processing method according to the embodiment of the present application.

[0113] In one embodiment, the program instructions are used to perform the following steps when running: obtaining an image to be processed; performing image processing on the image to be processed using a trained student processing model to obtain a processing result of the image to be processed, wherein the image processing includes three-dimensional target detection and / or three-dimensional instance segmentation, and the processing result includes a target detection result and / or an instance segmentation result; the trained student processing model is obtained by training the following operations: obtaining a first sample image and corresponding first annotation information, wherein the first annotation information is used to indicate the three-dimensional position of the target object contained in the first sample image; inputting the first sample image into the teacher encoding module of the initial teacher processing model for encoding, and obtaining a first teacher encoding feature; obtaining a first initial teacher query feature, wherein the first initial teacher query feature includes at least one at least one feature vector corresponding to each potential target object; merging the first initial teacher query feature with the first real query feature to obtain the first teacher query feature, wherein the first real query feature is obtained by position encoding based on the first annotation information; inputting the first teacher encoding feature and the first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding to obtain the first teacher processing result of the first sample image, the first teacher processing result including the target detection result and / or the instance segmentation result; training the initial teacher processing model based on at least the first teacher processing result and the first annotation information to obtain a trained teacher processing model; and performing distillation training on the first initial student processing model using the trained teacher processing model to obtain a trained student processing model.

[0114] In addition, according to an embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, and the computer program is used to execute the above-mentioned image processing method 200 when running.

[0115] Each module in the electronic device according to the embodiment of the present application can be implemented by running computer program instructions stored in a memory by a processor of an electronic device that implements image processing or image processing according to the embodiment of the present application, or can be implemented when computer instructions stored in a computer-readable storage medium of a computer program product according to the embodiment of the present application are executed by a computer.

[0116] In addition, according to an embodiment of the present application, a computer program is also provided, which is used to execute the above-mentioned image processing method 200 when running.

[0117] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present application. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not performed.

[0120] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0121] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to the present application should not be interpreted as reflecting the intention that the application claimed for protection requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features that are less than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0122] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0123] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0124] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the image processing apparatus according to the embodiments of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0125] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0126] The above is merely a description of specific embodiments of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, comprising: Get the image to be processed; Performing image processing on the image to be processed using the trained student processing model to obtain a processing result of the image to be processed, wherein the image processing includes three-dimensional object detection and / or three-dimensional instance segmentation, and the processing result includes an object detection result and / or an instance segmentation result; The trained student processing model is trained by the following training operations: Acquire a first sample image and corresponding first annotation information, where the first annotation information is used to indicate a three-dimensional position of a target object contained in the first sample image; Inputting the first sample image into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature; Acquire a first initial teacher query feature, where the first initial teacher query feature includes at least one feature vector corresponding to at least one potential target object; Merging the first initial teacher query feature with the first real query feature to obtain a first teacher query feature, wherein the first real query feature is obtained by position encoding based on the first annotation information; Inputting the first teacher encoding feature and the first teacher query feature into the teacher decoding module of the initial teacher processing model for decoding, to obtain a first teacher processing result of the first sample image, where the first teacher processing result includes an object detection result and / or an instance segmentation result; Training the initial teacher processing model based at least on the first teacher processing result and the first annotation information to obtain a trained teacher processing model; The trained teacher processing model is used to perform distillation training on the first initial student processing model to obtain the trained student processing model.

2. The method according to claim 1, wherein Before performing distillation training on the first initial student processing model using the trained teacher processing model to obtain the trained student processing model, the training operation further includes: Acquire a second sample image and corresponding second annotation information, where the second annotation information is used to indicate a three-dimensional position of a target object contained in the second sample image; Inputting the second sample image into the student encoding module of the second initial student processing model for encoding to obtain a first student encoding feature; Acquire a first student query feature, where the first student query feature includes at least one feature vector corresponding to at least one potential target object; Inputting the first student encoding feature and the first student query feature into a student decoding module of the second initial student processing model for decoding, to obtain a first student processing result of the second sample image, where the first student processing result includes an object detection result and / or an instance segmentation result; Based at least on the first student processing result and the second annotation information, the second initial student processing model is trained to obtain the first initial student processing model.

3. The method according to claim 2, wherein: The teacher encoding module includes a feature extraction module and a position encoder module. The first sample image is input into the teacher encoding module of the initial teacher processing model for encoding to obtain a first teacher encoding feature, including: Inputting the first sample image into the feature extraction module of the initial teacher processing model to perform feature extraction to obtain first teacher image features; Acquire a first position embedded feature corresponding to the first sample image; Inputting the first teacher image feature and the first position embedding feature into the position encoder module of the initial teacher processing model for position encoding to obtain the first teacher encoding feature; The student encoding module includes a feature extraction module and a position encoder module. The second sample image is input into the student encoding module of the second initial student processing model for encoding to obtain the first student encoding feature, including: Inputting the second sample image into the feature extraction module of the second initial student processing model to perform feature extraction to obtain first student image features; Acquire a second position embedded feature corresponding to the second sample image; The first student image feature and the second position embedding feature are input into the position encoder module of the second initial student processing model for position encoding to obtain the first student coding feature; wherein the feature extraction module of the initial teacher processing model and the feature extraction module of the second initial student processing model are the same shared feature extraction module.

4. The method according to claim 2, wherein: The teacher decoding module includes a decoder module and a processing head. The processing head of the teacher decoding module includes a detection head for outputting the target detection result and / or a segmentation head for outputting the instance segmentation result. The first teacher encoding feature and the first teacher query feature are input into the teacher decoding module of the initial teacher processing model for decoding to obtain the first teacher processing result of the first sample image, including: Inputting the first teacher encoding feature and the first teacher query feature into a decoder module of the initial teacher processing model to obtain a first teacher decoding feature; Inputting the first teacher decoding feature into the processing head of the initial teacher processing model to obtain the first teacher processing result; The student decoding module includes a decoder module and a processing head. The processing head of the student decoding module includes a detection head for outputting the target detection result and / or a segmentation head for outputting the instance segmentation result. The first student encoding feature and the first student query feature are input into the student decoding module of the second initial student processing model for decoding to obtain the first student processing result of the second sample image, including: Inputting the first student encoding feature and the first student query feature into a decoder module of the second initial student processing model to obtain a first student decoding feature; Inputting the first student decoding feature into the processing head of the second initial student processing model to obtain the first student processing result; The processing head of the initial teacher processing model and the processing head of the second initial student processing model are the same shared processing head.

5. The method according to claim 3 or 4, wherein: The second sample image is the same as the first sample image, wherein: In the process of training the initial teacher processing model based at least on the first teacher processing result and the first annotation information, a first prediction loss is determined based on the first teacher processing result and the first annotation information. In the process of training the second initial student processing model based at least on the first student processing result and the second annotation information, a second prediction loss is determined based on the first student processing result and the second annotation information. The initial teacher processing model and the second initial student processing model are synchronously trained based on a first total loss, wherein the first total loss is obtained based on the first prediction loss and the second prediction loss.

6. The method according to any one of claims 1 to 4, wherein: The step of performing distillation training on the first initial student processing model using the trained teacher processing model to obtain the trained student processing model comprises: acquiring a third sample image; Inputting the third sample image into the teacher encoding module of the trained teacher processing model for encoding to obtain a second teacher encoding feature; Acquire a second initial teacher query feature, where the second initial teacher query feature includes at least one feature vector corresponding to at least one potential target object; Merging the second initial teacher query feature with the second real query feature to obtain a second teacher query feature, wherein the second real query feature is obtained by position encoding based on third annotation information, and the third annotation information is used to indicate the three-dimensional position of the target object contained in the third sample image; Inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model for decoding to obtain the second teacher processing result of the third sample image, the second teacher processing result including the target detection result and / or the instance segmentation result, wherein the teacher decoding module includes a decoder module and a processing head, and inputting the second teacher encoding feature and the second teacher query feature into the teacher decoding module of the trained teacher processing model for decoding to obtain the second teacher processing result of the third sample image includes: inputting the second teacher encoding feature and the second teacher query feature into the decoder module of the trained teacher processing model to obtain the second teacher decoding feature; inputting the second teacher decoding feature into the processing head of the trained teacher processing model to obtain the second teacher processing result; Inputting the third sample image into the student encoding module of the first initial student processing model for encoding to obtain a second student encoding feature; Get the second student query feature; Inputting the second student encoding feature and the second student query feature into the student decoding module of the first initial student processing model for decoding to obtain the second student processing result of the third sample image, the second student processing result including the target detection result and / or the instance segmentation result, wherein the student decoding module includes a decoder module and a processing head, and inputting the second student encoding feature and the second student query feature into the student decoding module of the first initial student processing model for decoding to obtain the second student processing result of the third sample image includes: inputting the second student encoding feature and the second student query feature into the decoder module of the first initial student processing model to obtain the second student decoding feature; inputting the second student decoding feature into the processing head of the first initial student processing model to obtain the second student processing result; determining a third prediction loss based on at least the second teacher decoding feature and the second student decoding feature; determining a second total loss based at least on the third predicted loss; Optimize the parameters in the first initial student processing model based on the second total loss to obtain the trained student processing model.

7. The method according to claim 6, wherein: The decoder module in the teacher decoding module and the decoder module in the student decoding module each include N attention layers, where N is an integer greater than 1; The step of inputting the second teacher encoding feature and the second teacher query feature into a decoder module of the trained teacher processing model to obtain a second teacher decoding feature comprises: For each current attention layer in the decoder module of the trained teacher processing model, inputting an input feature into the current attention layer to obtain an output feature, wherein, when the current attention layer is the frontmost attention layer among the N attention layers, the input feature includes the second teacher encoding feature and the second teacher query feature; when the current attention layer is any attention layer other than the frontmost attention layer among the N attention layers, the input feature includes the second teacher encoding feature and the output feature of the previous attention layer, and the output feature of the last attention layer is the second teacher decoding feature; The step of inputting the second student encoding feature and the second student query feature into a decoder module of the first initial student processing model to obtain the second student decoding feature comprises: For each current attention layer in the decoder module of the first initial student processing model, inputting input features into the current attention layer to obtain output features, wherein, when the current attention layer is the frontmost attention layer among the N attention layers, the input features include the second student encoding features and the second student query features; when the current attention layer is any attention layer other than the frontmost attention layer among the N attention layers, the input features include the second student encoding features and the output features of the previous attention layer, and the output features of the last attention layer are the second student decoding features; The determining of a third prediction loss based at least on the second teacher decoding feature and the second student decoding feature comprises: Calculate the i-th sub-prediction loss based on the output features of the i-th attention layer included in the decoder module of the trained teacher processing model and the output features of the i-th attention layer included in the decoder module of the first initial student processing model, where i=1, 2, 3, ..., N; The calculated N sub-prediction losses are summed or averaged to obtain the third prediction loss.

8. An electronic device comprising a processor and a memory, wherein: The memory stores computer program instructions, which are used by the processor to execute the image processing method according to any one of claims 1 to 7 when the processor runs the computer program instructions.

9. A storage medium having program instructions stored thereon, wherein: The program instructions are used to execute the image processing method according to any one of claims 1 to 7 when running.

10. A computer program product comprising a computer program, wherein: The computer program is used to execute the image processing method according to any one of claims 1 to 7 when running.

Images