Image processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image processing method and device, electronic equipment and storage medium, the method comprising: obtaining an anchor point and first feature information corresponding to the anchor point from a feature map of an image to be processed, and generating at least one scattered point and second feature information corresponding to the at least one scattered point according to the anchor point and the first feature information corresponding to the anchor point, and then determining a processing result of a target image processing task according to the second feature information corresponding to the at least one scattered point. The method of the present disclosure is applied to a general perception module, can be embedded in image processing networks of different structures, and can perform different target image processing tasks, thereby realizing the generalization of the image processing network framework and the generalization of image processing tasks, and facilitating the reduction of design cost and migration cost of different task models.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to an image processing method and apparatus, electronic device and storage medium. Background Technology

[0002] Image classification, object detection, instance segmentation, and pose estimation are important components of computer vision and have significant applications in many fields of real-world production and daily life, such as smart cities, smart homes, and autonomous driving.

[0003] However, the network architecture designs for different image processing tasks are mostly complex and vary. In the process of adjusting and improving the parameters of the network architecture, a certain amount of experience in the corresponding field is required. In addition, since too much task-related prior knowledge is introduced into the network architecture design, progress in one task is difficult to be directly applied to another task, resulting in a relatively limited scope of application. Summary of the Invention

[0004] This disclosure proposes an image processing technology solution.

[0005] According to one aspect of this disclosure, an image processing method is provided, the method being applied to a general perception module, comprising: acquiring a feature map of an image to be processed; acquiring anchor points and first feature information corresponding to the anchor points from the feature map; generating at least one scattered point and second feature information corresponding to the at least one scattered point based on the anchor points and the first feature information corresponding to the anchor points; and determining the processing result of a target image processing task based on the second feature information corresponding to the at least one scattered point.

[0006] In one possible implementation, the target image processing task includes at least one of image classification, object detection, instance segmentation, and pose estimation.

[0007] In one possible implementation, generating at least one scattered point and at least one second feature information corresponding to the anchor point based on the anchor point and the first feature information corresponding to the anchor point includes: generating at least one scattered point based on the anchor point and the first feature information corresponding to the anchor point; and performing bilinear interpolation on the at least one scattered point to generate the second feature information corresponding to the at least one scattered point.

[0008] In one possible implementation, generating at least one scattering point based on the anchor point and the first feature information corresponding to the anchor point includes: inputting the first feature information corresponding to the anchor point into a first multilayer perceptron for processing to obtain at least one offset parameter of the anchor point; and generating at least one scattering point based on the anchor point, the at least one offset parameter, and at least one weight corresponding to the at least one offset parameter.

[0009] In one possible implementation, the location of the at least one scattering point is located in at least one of the following: for an image classification task, the at least one scattering point is located in the vicinity of the anchor point, and the distance between the scattering point generated in the vicinity and the anchor point is less than a preset distance; for an object detection task, the at least one scattering point is located in the four directions of up, down, left, and right of the anchor point; for an instance segmentation task, the at least one scattering point is located in the boundary region of the instance corresponding to the anchor point; for a human pose estimation task, the at least one scattering point is located at the average position of the human keypoints corresponding to the anchor point.

[0010] In one possible implementation, obtaining the anchor point from the feature map and the first feature information corresponding to the anchor point includes: obtaining the anchor point of the feature map, the anchor point including the center point of the instance in the feature map; and generating the first feature information corresponding to the anchor point based on the coordinates of the anchor point and / or the candidate region containing the anchor point.

[0011] In one possible implementation, determining the processing result of the target image processing task based on the second feature information corresponding to the at least one scattering point includes: concatenating the second feature information corresponding to the at least one scattering point and the task embedding vector to obtain a concatenated result, wherein the task embedding vector is used to characterize the type of the target image processing task; and inputting the concatenated result into an encoder for encoding to obtain the processing result of the target image processing task.

[0012] According to one aspect of this disclosure, an image processing apparatus is provided, the apparatus being applied to a general sensing module, comprising: a first acquisition module for acquiring a feature map of an image to be processed; a second acquisition module for acquiring anchor points and first feature information corresponding to the anchor points from the feature map; a generation module for generating at least one scattered point and second feature information corresponding to the at least one scattered point based on the anchor points and the first feature information corresponding to the anchor points; and a determination module for determining the processing result of a target image processing task based on the second feature information corresponding to the at least one scattered point.

[0013] In one possible implementation, the target image processing task includes at least one of image classification, object detection, instance segmentation, and pose estimation.

[0014] In one possible implementation, the generation module is configured to: generate at least one scattered point based on the anchor point and the first feature information corresponding to the anchor point; and perform bilinear interpolation on the at least one scattered point to generate second feature information corresponding to the at least one scattered point.

[0015] In one possible implementation, generating at least one scattering point based on the anchor point and the first feature information corresponding to the anchor point includes: inputting the first feature information corresponding to the anchor point into a first multilayer perceptron for processing to obtain at least one offset parameter of the anchor point; and generating at least one scattering point based on the anchor point, the at least one offset parameter, and at least one weight corresponding to the at least one offset parameter.

[0016] In one possible implementation, the location of the at least one scattering point is located in at least one of the following: for an image classification task, the at least one scattering point is located in the vicinity of the anchor point, and the distance between the scattering point generated in the vicinity and the anchor point is less than a preset distance; for an object detection task, the at least one scattering point is located in the four directions of up, down, left, and right of the anchor point; for an instance segmentation task, the at least one scattering point is located in the boundary region of the instance corresponding to the anchor point; for a human pose estimation task, the at least one scattering point is located at the average position of the human keypoints corresponding to the anchor point.

[0017] In one possible implementation, the second acquisition module is configured to: acquire anchor points of the feature map, the anchor points including the center points of instances in the feature map; and generate first feature information corresponding to the anchor points based on the coordinates of the anchor points and / or candidate regions containing the anchor points.

[0018] In one possible implementation, the determining module is used to: concatenate the second feature information corresponding to the at least one scatter point and the task embedding vector to obtain a concatenation result, wherein the task embedding vector is used to characterize the type of the target image processing task; and input the concatenation result into an encoder for encoding to obtain the processing result of the target image processing task.

[0019] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0020] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0021] In this embodiment, the general perception module can obtain anchor points and corresponding first feature information from the feature map of the image to be processed. Based on the anchor points and their corresponding first feature information, it generates at least one scattered point and corresponding second feature information. Then, based on the second feature information corresponding to the at least one scattered point, it determines the processing result of the target image processing task. This general perception module can be embedded in image processing networks with different structures to perform different target image processing tasks, such as image classification, object detection, instance segmentation, and pose estimation. It achieves a general image processing network framework and a general image processing task, which helps reduce the design and migration costs of different task models.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0024] Figure 1 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown.

[0025] Figure 2 The diagram shows the perception modules corresponding to different target image processing tasks in the related technologies.

[0026] Figure 3 A schematic diagram of a general sensing module according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A schematic diagram of an image processing network according to an embodiment of the present disclosure is shown.

[0028] Figure 5 A schematic diagram illustrating the generation of scattered points according to an embodiment of the present disclosure is shown.

[0029] Figure 6 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0030] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0031] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0033] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0034] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0035] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0036] Figure 1 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the image processing method includes:

[0037] In step S11, the feature map of the image to be processed is obtained;

[0038] In step S12, anchor points and the first feature information corresponding to the anchor points are obtained from the feature map;

[0039] In step S13, at least one scattered point and at least one scattered point corresponding to a first feature information are generated based on the anchor point and the first feature information corresponding to the anchor point.

[0040] In step S14, the processing result of the target image processing task is determined based on the second feature information corresponding to the at least one scattered point.

[0041] In one possible implementation, the image processing method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0042] In one possible implementation, the image processing method can be implemented by a processor calling computer-readable instructions stored in memory. In one example, the processor can be a general-purpose processor such as a central processing unit (CPU), a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC), or it can be an artificial intelligence processor, such as an artificial intelligence (AI) chip, for example, a neural processing unit (NPU).

[0043] In one possible implementation, the acquired image to be processed can be input into a trained image processing network, which will then perform image processing on the image and output the processing result.

[0044] For example, the image to be processed can be any image acquired by the electronic device. The electronic device can acquire images of the current scene to obtain one or more images to be processed. Alternatively, the electronic device can select one or more image frames as the images to be processed from multiple image frames included in a video file. In some implementations, the electronic device can acquire one or more images to be processed from other devices. This disclosure does not limit this.

[0045] For example, the image processing network can be any network embedded with a general-purpose sensing module. The image processing network can include at least one of the following: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Neural Networks (DNN), YOLO (You Only Look Once) network based on deep learning, Residual Networks (ResNets), Back Propagation Neural Networks (BP), Backbone Neural Networks, etc.

[0046] For example, an image processing network may include a backbone network, a neck network, and a general perception module. The backbone network and neck network are used to extract features from the image to be processed, obtaining feature maps of the image; the general perception module is used to determine the processing result for the corresponding target image processing task based on the obtained feature maps.

[0047] The backbone network can be used for feature extraction to extract features of different scales, receptive fields, and categories. It can include VGG networks in convolutional neural networks, ResNet residual networks (e.g., ResNet18, 50, 100), densely connected convolutional networks (DenseNet), lightweight neural networks (SqueezeNet), lightweight networks (MobileNet), deterministic networks (DetNet), and search object detection backbone networks (DetNAS). The neck network can be used to fuse and process features extracted from multiple levels by the backbone network. It can include Spatial Pyramid Pooling (SPP), Atrous Spatial Pyramid Pooling (ASPP), and Receptive Fields Block (RFB).

[0048] It should be understood that the network structure of the above-described image processing network is only an example. When the image processing network includes a general perception module, it may include multiple convolutional layers, multiple pooling layers, multiple fully connected layers, etc. The specific construction method and structure of the network can be determined according to the actual situation. The above example does not constitute a limitation on the embodiments of this disclosure.

[0049] In one possible implementation, the general perception module is used to embed image processing networks with different structures to perform different target image processing tasks. These target image processing tasks include at least one of image classification, object detection, instance segmentation, and pose estimation.

[0050] To better illustrate the general sensing module of the embodiments of this disclosure, the sensing module in related technologies will be introduced first.

[0051] In related technologies, due to the introduction of too much task-related prior knowledge in the structural design of image processing networks, progress in a single task is difficult to be directly applied to another task, exhibiting task exclusivity. For example, a perception module developed in an object detection task cannot be transferred to a pose estimation task. Different perception modules need to be designed for different target image processing tasks (such as image classification, object detection, instance segmentation, and pose estimation).

[0052] Figure 2 This diagram illustrates the perception modules corresponding to different target image processing tasks in related technologies. For example... Figure 2 As shown, different target image processing tasks correspond to features with different structures, and different target image processing tasks also correspond to prediction results in different forms. For example, the prediction result output by the image classification task is the category label; the prediction result output by the object detection task is the prediction box and the prediction category; the prediction result output by the instance segmentation task is the pixelated mask; the prediction result of the pose estimation task is key points, feature points, etc. Furthermore, the prediction result of the pose estimation task can also include pose type, action type, etc.

[0053] It is evident that in related technologies, different perception modules need to be designed for different target image processing tasks and / or image processing networks with different structures, such as... Figure 2 As shown, the perception module 21 for image classification, the perception module 22 for object detection, the perception module 23 for instance segmentation, and the perception module 24 for pose estimation are different. Therefore, for a new target image processing task, and / or a new image processing network structure, the perception modules of other tasks cannot be directly applied; a new perception module needs to be designed, which incurs significant design and migration costs.

[0054] Conversely, the general-purpose sensing module of this disclosure can be embedded in image processing networks with different structures to perform different target image processing tasks. It can be easily extended to different target image processing tasks or image processing networks with different structures, and can accurately determine the processing result in any target image processing task.

[0055] Figure 3A schematic diagram of a general sensing module according to an embodiment of the present disclosure is shown. For example... Figure 3 As shown, the general perception module can be embedded in image processing networks with different structures, and can achieve different target image processing tasks through multi-point estimation. For example, the general perception module can obtain anchor points from the feature map and adaptively scatter them into different spaces to generate multiple scattered points; then, the general perception module can use a stacked encoder (e.g., an encoder formed by stacking multiple Transformer encoders) to analyze the interrelationships between the multiple scattered points, and output multiple predicted points in a multi-point form. Due to this multi-point form, it is robust to geometric changes (e.g., including scale, deformation, and orientation) exhibited by the target object. Therefore, based on these predicted points, processing results corresponding to different target image processing tasks can be obtained.

[0056] For example, in image classification tasks, these prediction points can be used to predict the category of the target object. For instance, these prediction points can be input into a multilayer perceptron to output the predicted category. In object detection tasks, these prediction points will be placed along the four sides of the prediction box. In instance segmentation tasks, these prediction points can be uniformly distributed along the contour of the instance mask. In pose estimation tasks, the positions of these points conform to the pose distribution of the training data.

[0057] In one possible implementation, for any image processing task, the image processing network with a general perception module embedded above can be used to obtain the processing result of the image to be processed.

[0058] Figure 4 A schematic diagram of an image processing network according to an embodiment of the present disclosure is shown. Figure 4 As shown, the image processing network may include a feature extraction module 41 composed of a skeleton network and a feature pyramid network, and a general perception module 42. The feature extraction module 41 is used to extract features from the image to be processed to obtain a feature map of the image. The general perception module 42 is used to determine the processing result of the target image processing task (e.g., image classification, object detection, instance segmentation, and pose estimation) based on the feature map.

[0059] For example, in step S11, the feature extraction module 41, which consists of a skeleton network and a feature pyramid, can extract features from the image to be processed to determine the feature map of the image to be processed. The feature map of the image to be processed determined by the feature extraction module 41 can be output to the general perception module 42, so that the general perception module 42 can acquire the feature map of the image to be processed.

[0060] It should be understood that this disclosure is for informational purposes only. Figure 4Taking the feature extraction module 41 as an example, the feature extraction module 41 can also be other neural network structures used to obtain feature maps, and this disclosure does not limit it. Furthermore, the feature map obtained by the feature extraction module 41 may include the texture features, shape features, spatial features, color features, edge features, key point features, etc. of the target object, and this disclosure does not limit the size or form of the feature map.

[0061] In step S11, the feature map of the image to be processed is obtained. In steps S12 to S14, the feature map of the image to be processed is processed by the general perception module 42 to determine the processing result of the image to be processed.

[0062] For example, in step S12, the general perception module can obtain the anchor point and the first feature information corresponding to the anchor point from the acquired feature map. For example, the anchor point can be the center point of the instance in the feature map, and the first feature information corresponding to the anchor point can include the anchoring ratio of the anchor point (that is, the ratio of the width and height of the anchor frame), the scale of the anchor point (that is, the width and height of the anchor frame), etc.

[0063] In step S13, at least one scattering point can be generated based on the anchor point and its corresponding first feature information. For example, scattering point learning can be performed based on the first feature information of the anchor point to obtain at least one offset information relative to the anchor point, such as... Figure 4 As shown by the arrows, each offset information can include horizontal offset information and vertical offset information. Then, based on the anchor point's position information (coordinates) and the offset information, at least one scatter point can be generated, and bilinear interpolation can be used to obtain the second feature information corresponding to the scatter point on the feature map.

[0064] The second feature information may include feature information corresponding to the scatter points on the feature map. For example, the second feature information may be a feature vector, which may include the coordinates of the scatter points (the coordinates of the scatter points may include a decimal part), the coordinates of at least one pixel point in the feature map corresponding to the coordinates of the scatter points (the coordinates of any pixel point in the feature map are integers, excluding the decimal part), and the pixel value at the coordinates of each pixel point in the feature map, etc. Alternatively, the second feature information may also be obtained by weighting the pixel value at the coordinates of the scatter points and the pixel points in the feature map. This disclosure does not limit the specific form of the second feature information.

[0065] In step S14, the second feature information corresponding to at least one scattered point can be concatenated with the learnable task embedding vector, and the concatenation result can be input into the Transformer encoder for encoding processing, and the processing result of the target image processing task can be output.

[0066] The Transformer encoder can be composed of a multi-headed self-attention module (MHSA) and a multilayer perceptron (MLP) (e.g., a two-layer point-oriented MLP). This disclosure does not limit the specific structure of the Transformer encoder.

[0067] The processing results can be category labels for image classification tasks, predicted bounding boxes and predicted categories for object detection tasks, pixelated masks for instance segmentation tasks, key points and feature points for pose estimation tasks, etc. The processing results for pose estimation tasks can also include pose type, action type, etc.

[0068] In this way, the general perception module can obtain anchor points and corresponding first feature information from the feature map of the image to be processed. Based on the anchor points and their corresponding first feature information, it generates at least one scattered point and corresponding second feature information. Then, based on the second feature information corresponding to the at least one scattered point, it determines the processing result of the target image processing task. The general perception module of this disclosure can be embedded in image processing networks with different structures to perform different target image processing tasks, such as image classification, object detection, instance segmentation, and pose estimation. It achieves the universality of the image processing network framework and the universality of the image processing tasks, which helps to reduce the design and migration costs of different task models.

[0069] The image processing method of the present disclosure embodiment will be described in detail below.

[0070] In step S11, the feature map of the image to be processed is obtained. In step S12, the anchor point and the first feature information corresponding to the anchor point are obtained from the feature map.

[0071] In one possible implementation, step S12 may include: obtaining anchor points of the feature map, the anchor points including the center points of instances in the feature map; and generating first feature information corresponding to the anchor points based on the coordinates of the anchor points and / or candidate regions containing the anchor points.

[0072] For example, the center point of an instance in the feature graph can be determined as an anchor point. Different objects of the same category in the feature graph can each correspond to an instance. If the feature graph includes multiple instances, each instance can be determined as a corresponding anchor point. This disclosure does not impose any restrictions on this.

[0073] Once the anchor point is determined, the first feature information corresponding to the anchor point can be generated. For example, for one-stage detection methods or anchor-free detection methods, the coordinates of the anchor point obtained from the feature map can be directly determined as the first feature information corresponding to the anchor point; for two-stage detection methods, the first feature information corresponding to the anchor point can be determined based on the candidate region containing the anchor point, for example, the first feature information corresponding to the anchor point can be generated based on the RoI Pooling method.

[0074] This method facilitates the efficient and accurate acquisition of anchor points and their primary feature information, which is beneficial for generating multiple scattered points based on the anchor points in the future.

[0075] In step S12, the anchor point and the first feature information corresponding to the anchor point are obtained. In step S13, at least one scattered point and at least one second feature information corresponding to the scattered point are generated based on the anchor point and the first feature information corresponding to the anchor point.

[0076] In one possible implementation, step S13 may include:

[0077] In step S131, at least one scattered point is generated based on the anchor point and the first feature information corresponding to the anchor point;

[0078] In step S132, bilinear interpolation is performed on the at least one scattered point to generate second feature information corresponding to the at least one scattered point.

[0079] In one possible implementation, in step S131, the first feature information corresponding to the anchor point can be input into the first multilayer perceptron for processing to obtain at least one offset parameter of the anchor point; at least one scattering point is generated based on the anchor point, the at least one offset parameter, and at least one weight corresponding to the at least one offset parameter.

[0080] For example, the generation of scatter points can be represented as follows:

[0081] P xi =A x +s x ·Δx i

[0082] P yi =A y +s y ·Δy i (1)

[0083] In formula (1), Represents the i-th generated scatter point, i∈[1,K]; (A xA y ) represents the anchor point, (Δx i ,Δy i ) represents the i-th offset parameter, which can be used to set the anchor point (A) x A y The first feature information F corresponding to ) x,y The input is processed by the first multilayer perceptron f to obtain the anchor point (A). x A y The K (K≥1) offset parameters of ) are: (Δx i ,Δy i )=f(F x,y ;w i ), f represents the first multilayer perceptron (MLP), w i The learnable parameters represent the first multilayer perceptron f;

[0084] In formula (1), (s x ,s y ) represents the offset parameter (Δx) i ,Δy i The weights of ) are used to adjust the offset parameter (Δx) i ,Δy i The size of (s); for example, for a one-stage detection method, (s x ,s y ) can be the dimensions of the anchor point, i.e., the width and height of the anchor frame; for the two-stage detection method, (s x ,s y ) can be the width and height of the region proposals (regions that may contain objects); for anchor-free detection, (s x ,s y () can be the model step size.

[0085] In this way, based on the anchor point (A) x A y The first feature information F x,y By performing scatterpoint learning, K offset parameters (Δx) are obtained. i ,Δy i ), i∈[1,K]; then according to the offset parameter (Δx) i ,Δy i ), and offset parameter (Δx) i ,Δy i The weights (s) x ,s y ), for anchor point (A) x A yBy performing offset processing, K scattered points can be generated efficiently and accurately. This multi-point form is robust to geometric changes exhibited by the target object (such as scale, deformation, and orientation), which is beneficial for generating processing results applicable to different image processing tasks.

[0086] Figure 5 A schematic diagram illustrating the generation of distributed points according to an embodiment of this disclosure is shown. Figure 5 As shown, for the same anchor point of the same image to be processed ( Figure 5 The black dots in sections 5a to 5d, from left to right, represent the generated images for image classification, object detection, instance segmentation, and pose estimation tasks, respectively. Figure 5 As shown in Part 5a), object detection tasks (such as...) Figure 5 As shown in section 5b), instance segmentation task (such as...) Figure 5 As shown in section 5c), pose estimation task (such as...) Figure 5 The dispersion points shown in section 5d) Figure 5 (White dots in parts 5a to 5d).

[0087] For image classification tasks, the at least one scattered point is located in the vicinity of the anchor point, and the distance between the scattered point generated in the vicinity and the anchor point is less than a preset distance.

[0088] For the target detection task, the at least one scattering point is located in the four directions of up, down, left, and right of the anchor point; wherein, a set of scattering points can be generated in each direction.

[0089] For instance segmentation tasks, the at least one scatter point is located in the boundary region of the instance corresponding to the anchor point; for example, a two-dimensional reference vector starting from the anchor point can be set according to the anchor point and the first feature information corresponding to the anchor point, and the direction of the vector can be determined accordingly. Figure 5 In section 5c (in the direction of the arrow), scattered points are uniformly generated on the edge of the pseudo-box (square) generated by the anchor point and its spatial scale, that is, the boundary region of the instance corresponding to the anchor point.

[0090] For the human pose estimation task, the at least one scattered point is located at the average position of the human keypoints corresponding to the anchor point. The human keypoints include keypoints of facial organs and / or limb keypoints, and the average position of the human keypoints can be obtained by statistically analyzing the training dataset, for example, by statistically analyzing the average position of the human keypoints labeled in each sample image of the training dataset.

[0091] As can be seen, the general perception module of this disclosure can adaptively initialize scattered points for different target image processing tasks, so as to adapt to processing objects of different scales and poses under different target image processing tasks.

[0092] In the above process, the offset parameter (Δx) can be adjusted by adjusting the bias (also known as the bias parameter) of the last fully connected layer in the first multilayer perceptron f. i ,Δy i The magnitude of ) is then determined based on the offset parameter (Δx). i ,Δy i The position of the generated scattered points can be controlled. For example, taking object detection as an example, the offset bias of the scattered points in each group (left, right, top, and bottom) can be set to [-0.5, 0], [0.5, 0], [0, -0.5], and [0, 0.5], respectively. This disclosure does not limit the specific value of the offset bias.

[0093] In this way, the initial position of the scattered points is controlled by adjusting the offset parameters in the fully connected layer of the first multilayer perceptron f. Different scattered points can be generated adaptively for different target image processing tasks, which reduces the learning difficulty on different target image processing tasks and improves the versatility of the scattered point learning method. It can be applied to different target image processing tasks of general visual perception, including image classification tasks, object detection tasks, instance segmentation tasks and pose estimation tasks.

[0094] K scattered points were generated in step S131, and in step S132, the K scattered points were... Perform bilinear interpolation to generate each scatter point The corresponding second feature information F xi,yi , i∈[1,K].

[0095] For example, suppose Let K be any of the K scattered points. Considering that the coordinates of this scattered point may contain decimal parts, while the coordinates of each pixel on the feature map are integers, there may not be a single pixel on the feature map that corresponds to an exact coordinate position.

[0096] In this case, the feature map can be accessed via coordinate positions. The weighted sum of the four surrounding pixels represents the scatter point, which is used to find the corresponding scatter point from the feature map. The coordinates of four adjacent pixels can be obtained by analyzing the scattered points. Bilinear interpolation is performed, that is, linear interpolation is performed in two directions (e.g., the x-axis and y-axis) to obtain a representation of the scatter points on the feature map. The four adjacent pixels.

[0097] In order to simplify calculations and improve processing efficiency, the scattered points can be... The x-axis and y-axis coordinates are rounded up and down respectively to obtain four adjacent coordinates, which correspond to the coordinate positions of the feature map. The coordinates of the four adjacent pixels.

[0098] Then, based on the coordinates of the scatter points... Scattering points The coordinates of four adjacent pixels in the corresponding feature map, and the pixel values ​​of those four adjacent pixels, are used to generate scatter points. The corresponding second feature information F xi,yi For example, you can directly use the coordinates of the scattered points. Scattering points The coordinates of four adjacent pixels in the corresponding feature map, along with their pixel values, are concatenated. The resulting feature vector is then used as the scattering point. The corresponding second feature information F xi,yi For example, it can also be done through the coordinates of scattered points. Scattering points The coordinates of four adjacent pixels in the corresponding feature map are used to weight the pixel values ​​of these four adjacent pixels, and the weighted result is used to determine the scatter points. The corresponding second feature information F xi,yi This disclosure does not limit the specific method of generating the second feature information.

[0099] In this way, bilinear interpolation can be used to map the scattered points (whose coordinates may contain decimal parts) to the pixels in the feature map (whose coordinates do not contain decimal parts), which is beneficial for efficiently and conveniently obtaining the feature information corresponding to the scattered points on the feature map.

[0100] Through steps S131 to S132, the anchor point (A) can be determined. x A y The first feature information F corresponding to the anchor point. x,y Generate K scattered points i∈[1,K], and the second feature information F corresponding to the K scattered points xi,yi The multi-point representation of i∈[1,K] is robust to geometric changes (such as scale, deformation and orientation) of the target object, which is beneficial for generating processing results applicable to different image processing tasks.

[0101] In step S13, second feature information corresponding to at least one scattered point is obtained. In step S14, the processing result of the target image processing task can be determined based on the second feature information corresponding to at least one scattered point.

[0102] In one possible implementation, step S14 may include:

[0103] In step S141, the second feature information corresponding to the at least one scattered point and the task embedding vector are concatenated to obtain a concatenation result. The task embedding vector is used to characterize the type of the target image processing task.

[0104] In step S142, the stitching result is input into the encoder for encoding processing to obtain the processing result of the target image processing task.

[0105] For example, in step S141, in order to improve robustness to different target image processing tasks, the second feature information F corresponding to the K scattered points is... xi,yi i∈[1,K], and a task embedding vector T used to characterize the type of the target image processing task. task After performing the splicing process, the spliced ​​result z0 is obtained, that is:

[0106] z0 = [T] task ;F x1,y1 ;F x2,y2 ;…;F xK,yK (2)

[0107] Wherein, the task embedding vector T task It may include the classification task embedding vector T class , Object detection task embedding vector T IoU Embedding vector T for pose estimation task visibility .

[0108] The stitching result obtained in step S141 can be input into the Transformer encoder for encoding in step S142 to obtain the processing result of the target image processing task, which can be expressed as:

[0109] z′ l =MHSA(LN(z) l-1 ))+z l-1 ,l=1…L

[0110] z l =MLP(LN(z′) l ))+z′ l ,l=1…L (3)

[0111] In formula (3), LN represents Layer Normalization (LN), MHSA represents the Multi-Head Self-Attention module of the Transformer encoder, MLP represents the Multilayer Perceptron (MLP) of the Transformer encoder, l represents the current l-th iteration process, l = 1…L, zl-1 This represents the input data of the Transformer encoder in the l-th round (current round), which is also the prediction result of the Transformer encoder in the (l-1)-th round (previous round), z′ l This represents the output data of the Multi-Head Self-Attention Module (MHSA) during the current l-th iteration, which is also the input data of the Multilayer Perceptron (MLP).

[0112] In this context, the input for the first round (l=1) of the Transformer encoder's iterative training is the concatenated result z0, and the output for the last round (l=L) of the Transformer encoder's iterative training is: z L =[T′ task ;F′ x1,y1 ;F′ x2,y2 ;…;F′ xK,yK ], z L It can be used to determine the processing results of target image processing tasks and can be adapted to different target image processing tasks for task-aware prediction.

[0113] In this way, by introducing learnable task embedding vectors during the process of using the Transformer encoder to process scattered point features, not only can the relationships between these scattered points be better explained, which is beneficial to improving the accuracy of the processing results, but also to adapting to various target image processing tasks.

[0114] The following analysis and explanation will describe the use of the general perception module of this disclosure to perform image classification tasks, object detection tasks, instance segmentation tasks, and pose estimation tasks.

[0115] For example, for image classification tasks, scatterpoint learning can be performed using feature maps of the image to be processed. The center of the feature map is determined as the anchor point, and the scale (length and width) of the feature map is determined as the first feature information corresponding to the anchor point. This allows the Transformers encoder to be consistent with the classifier used to perform the image classification task; that is, the Transformers encoder only uses the classifier's classification labels, rather than using all the labels in the classifier. Thus, the classification loss function L used to perform the image classification task... cls It can be represented as:

[0116] L cls =CrossEntropy(softmax(MLP(T′ cls )),y) (4)

[0117] In formula (4), y represents the true class used as the evaluation criterion (e.g., the class labeled in the training dataset), and T′ clsrepresents the predicted classification task embedding vector output by the Transformers encoder; MLP stands for Multilayer Perceptron, which can include a single fully connected layer; softmax represents the normalized exponential function, and CrossEntropy represents the cross-entropy function.

[0118] For example, for object detection tasks, the general perception module can be applied to various object detection networks, such as Faster R-CNN (Fast Region Convolutional Neural Network) and Fully Convolutional First-Order Object Detection Network (FCOS). The embedded general perception module does not change the backbone network structure or label assignment method of the object detection network. During the execution of the object detection task, the object detection task embedding vector T... IoU It can replace the IoU branch of an object detection network. After the scattered points are encoded by the Transformers encoder, the predicted object detection task embedding vector T′ output by the Transformers encoder can be used. Iou By predicting the IoU and combining it with the classification prediction described above in the image classification task, the processing result of the object detection task is obtained. The F′ output by the Transformers encoder... xi,yi It can be used to predict scatter points The offset, that is:

[0119]

[0120] Among them, (s x ,s y ) represents the function used to adjust F′ xi,yi The weights of the size, ⊙ represents element-wise multiplication, MLP stands for Multilayer Perceptron, which can include fully connected layers shared between different points. This represents the adjusted dispersion points.

[0121] Thus, the prediction box can be represented as

[0122] For the classification branch, its calculation method is the same as that for the image classification task described above, and will not be repeated here. During the regression process, the IoU branch can share the concatenation result z0 with the classification branch to reduce the computational cost of generating multi-point representations.

[0123] Thus, the target detection loss function used to perform the target detection task can be expressed as:

[0124]

[0125] Where j represents the index number, n represents the total number of indices, and B′j B represents the j-th prediction box. j L1 represents the j-th ground truth box (the labeled box of the sample image in the training dataset), and L1 represents the mean absolute error loss function. Other types of detection loss can also be used, such as the generalized intersection over union (GIoU) ​​loss, etc. This disclosure does not limit this.

[0126] For example, for instance segmentation tasks, the prediction process can be viewed as contour-based regression. The general perception module can be connected to the output of the skeleton network and generate the prediction using equations (1) to (3) and equation (5). To align the labeled contour points in the training dataset with the scattered points, new points can be added evenly, or existing points can be deleted, until the target number is met. This is similar to the real-time instance segmentation method (DeepSnake) that combines deep learning and the Snake active contour model. In this way, the ground truth points (the labeled contour points in the training dataset) can be arranged clockwise around the centroid of all points, and then the predicted points (the adjusted scattered points) are... They are also arranged clockwise around this centroid, thus forming a one-to-one correspondence with the real points.

[0127] In this case, some target objects in the image to be processed are divided into several parts due to occlusion. To reduce this situation, they can be treated as multiple objects for detection.

[0128] Thus, the instance segmentation loss function used to perform instance segmentation tasks can be expressed as:

[0129]

[0130] Where n represents the total number of predicted output points. Represents the i-th prediction point. L1 represents the i-th labeled point, and L1 represents the mean absolute error loss function.

[0131] For example, for the pose estimation task, the instance segmentation task can be described above, and the pose estimation task embedding vector T can be introduced. visibility The visibility of keypoints is predicted. The number of predicted points, K, is aligned with the keypoints in the training dataset. For pose estimation tasks, each keypoint can correspond to a well-defined term, such as the nose or eyes, making it possible to establish a one-to-one correspondence between keypoints and predicted points. The localization branch of the keypoints can be trained using the L1 loss method (mean absolute error loss), as shown in formula (7). For training the keypoint visibility prediction, the standard binary cross-entropy loss function can be used.

[0132] As can be seen, the image processing method of this disclosure can more easily implement the training of multiple target image processing tasks, and can complete the joint training of multiple target image processing tasks to obtain a general visual perception model. This is beneficial to reducing the design cost and transfer cost of different task models and improving the efficiency of parameter adjustment. Moreover, using the processing results of this disclosure, the automatic annotation of training data required for multiple tasks can be completed simultaneously.

[0133] In summary, according to the embodiments of this disclosure, a feature map of the image to be processed can be obtained, anchor points and corresponding first feature information can be obtained from the feature map, and at least one scattered point and corresponding second feature information can be generated based on the anchor points and the corresponding first feature information. Then, the processing result of the target image processing task can be determined based on the second feature information corresponding to the at least one scattered point. The general perception module of the embodiments of this disclosure can be applied not only to various visual tasks but also to various visual perception frameworks. It also achieves the universality of image processing network frameworks and image processing tasks, which helps to reduce the design cost and migration cost of different task models and makes it easier to train various visual tasks.

[0134] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0135] In addition, this disclosure also provides an image processing apparatus, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the image processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0136] Figure 6 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown, such as Figure 6 As shown, the device is applied to a general sensing module, and the device includes:

[0137] The first acquisition module 61 is used to acquire the feature map of the image to be processed;

[0138] The second acquisition module 62 is used to acquire anchor points from the feature map and the first feature information corresponding to the anchor points;

[0139] The generation module 63 is used to generate at least one scattered point and at least one scattered point corresponding to a second feature information based on the anchor point and the first feature information corresponding to the anchor point.

[0140] The determination module 64 is used to determine the processing result of the target image processing task based on the second feature information corresponding to the at least one scattered point.

[0141] In one possible implementation, the target image processing task includes at least one of image classification, object detection, instance segmentation, and pose estimation.

[0142] In one possible implementation, the generation module 63 is configured to: generate at least one scattered point based on the anchor point and the first feature information corresponding to the anchor point; and perform bilinear interpolation on the at least one scattered point to generate second feature information corresponding to the at least one scattered point.

[0143] In one possible implementation, generating at least one scattering point based on the anchor point and the first feature information corresponding to the anchor point includes: inputting the first feature information corresponding to the anchor point into a first multilayer perceptron for processing to obtain at least one offset parameter of the anchor point; and generating at least one scattering point based on the anchor point, the at least one offset parameter, and at least one weight corresponding to the at least one offset parameter.

[0144] In one possible implementation, the location of the at least one scattering point is located in at least one of the following: for an image classification task, the at least one scattering point is located in the vicinity of the anchor point, and the distance between the scattering point generated in the vicinity and the anchor point is less than a preset distance; for an object detection task, the at least one scattering point is located in the four directions of up, down, left, and right of the anchor point; for an instance segmentation task, the at least one scattering point is located in the boundary region of the instance corresponding to the anchor point; for a human pose estimation task, the at least one scattering point is located at the average position of the human keypoints corresponding to the anchor point.

[0145] In one possible implementation, the second acquisition module 62 is configured to: acquire anchor points of the feature map, the anchor points including the center points of instances in the feature map; and generate first feature information corresponding to the anchor points based on the coordinates of the anchor points and / or candidate regions containing the anchor points.

[0146] In one possible implementation, the determining module 64 is used to: concatenate the second feature information corresponding to the at least one scattering point and the task embedding vector to obtain a concatenation result, wherein the task embedding vector is used to characterize the type of the target image processing task; and input the concatenation result into an encoder for encoding to obtain the processing result of the target image processing task.

[0147] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0148] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0149] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0150] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0151] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0152] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0153] Figure 7 This diagram illustrates a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal devices.

[0154] Reference Figure 7 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0155] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0156] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0157] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0158] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0159] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0160] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0161] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0162] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, LTE, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID), Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0163] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0164] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0165] Figure 8 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0166] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0167] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0168] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0169] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0170] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0171] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0172] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0173] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0174] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0176] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0177] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0178] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0179] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0180] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that, The method is applied to a general sensing module, and the method includes: Obtain the feature map of the image to be processed; Anchor points and corresponding first feature information are obtained from the feature map; Based on the anchor point and the first feature information corresponding to the anchor point, at least one scattered point and the second feature information corresponding to the at least one scattered point are generated. For different target image processing tasks, different scattered points are adaptively generated. The at least one scattered point is generated by scattering the anchor point into different spaces. Based on the second feature information corresponding to the at least one scattered point, the processing result of the target image processing task is determined; The step of determining the processing result of the target image processing task based on the second feature information corresponding to the at least one scattered point includes: The second feature information corresponding to the at least one scattered point and the task embedding vector are concatenated to obtain a concatenation result, wherein the task embedding vector is used to characterize the type of the target image processing task. The stitching result is input into the encoder for encoding processing to obtain the processing result of the target image processing task.

2. The method according to claim 1, characterized in that, The target image processing task includes at least one of image classification, object detection, instance segmentation, and pose estimation.

3. The method according to claim 1, characterized in that, The step of generating at least one scattered point and at least one scattered point corresponding to a second feature information based on the anchor point and the first feature information corresponding to the anchor point includes: Based on the anchor point and the first feature information corresponding to the anchor point, at least one scattered point is generated; Bilinear interpolation is performed on the at least one scattered point to generate second feature information corresponding to the at least one scattered point.

4. The method according to any one of claims 1-3, characterized in that, The step of generating at least one scattered point based on the anchor point and the first feature information corresponding to the anchor point includes: The first feature information corresponding to the anchor point is input into the first multilayer perceptron for processing to obtain at least one offset parameter of the anchor point. At least one scattering point is generated based on the anchor point, the at least one offset parameter, and the at least one weight corresponding to the at least one offset parameter.

5. The method according to any one of claims 1-4, characterized in that, The location of the at least one dispersion point is located in at least one of the following: For image classification tasks, the at least one scattered point is located in the vicinity of the anchor point, and the distance between the scattered point generated in the vicinity and the anchor point is less than a preset distance; For the target detection task, the at least one scattered point is located in the four directions of up, down, left, and right of the anchor point; For instance segmentation tasks, the at least one scattering point is located in the boundary region of the instance corresponding to the anchor point; For the human posture estimation task, the at least one scattered point is located at the average position of the human key points corresponding to the anchor point.

6. The method according to any one of claims 1-5, characterized in that, The step of obtaining the anchor point from the feature map and the first feature information corresponding to the anchor point includes: Obtain the anchor points of the feature map, wherein the anchor points include the center points of instances in the feature map; Based on the coordinates of the anchor point and / or the candidate region containing the anchor point, generate the first feature information corresponding to the anchor point.

7. An image processing apparatus, characterized in that, The device is applied to a general sensing module and includes: The first acquisition module is used to acquire the feature map of the image to be processed; The second acquisition module is used to acquire anchor points from the feature map and the first feature information corresponding to the anchor points; The generation module is used to generate at least one scattered point and at least one second feature information corresponding to the anchor point based on the anchor point and the first feature information corresponding to the anchor point. The at least one scattered point is generated by scattering the anchor point into different spaces for different target image processing tasks. The determination module is used to determine the processing result of the target image processing task based on the second feature information corresponding to the at least one scattered point; The determining module is used for: The second feature information corresponding to the at least one scattered point and the task embedding vector are concatenated to obtain a concatenation result, wherein the task embedding vector is used to characterize the type of the target image processing task. The stitching result is input into the encoder for encoding processing to obtain the processing result of the target image processing task.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images