Image processing method, model, device and equipment, and storage medium

Abstract

The invention provides an image processing method and device, equipment and a storage medium, and the method comprises the steps: responding to a to-be-processed image, and extracting a first feature map corresponding to the to-be-processed image through a first model; under the condition that a target object exists in the first feature map, performing image recognition on the to-be-processed image and / or the feature map by using a second model so as to output a recognition result containing the target object; wherein the first model and the second model belong to different sub-models of the same model.

Description

Technical Field

[0001] This application relates to the field of image detection, and more particularly to an image processing method, apparatus and device, and storage medium. Background Technology

[0002] In the process of detecting and locating target objects in an image, the entire detection and localization process must be implemented regardless of whether the target object exists in the image, which leads to low efficiency in detecting and locating target objects in the image. Summary of the Invention

[0003] This application provides an image processing method, apparatus, device, and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this application, an image processing method is provided, the method comprising:

[0005] In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model;

[0006] If a target object is present in the first feature map, the second model is used to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object;

[0007] The first model and the second model are different sub-models of the same model.

[0008] In one possible implementation, the first model includes a convolutional neural network, and the step of extracting a first feature map corresponding to the image to be processed using the first model includes:

[0009] The image to be processed is input into the first model, and the basic image features of the image to be processed are extracted through the convolutional neural network of the first model;

[0010] The first feature map is generated based on the image's basic features; the image's basic features represent at least one of pixel grayscale, color features, contrast, edge features, and texture features.

[0011] In one possible implementation, the method further includes:

[0012] The first feature map is pooled using the detection and classification head of the first model to obtain the importance weights of the feature channels of the first feature map.

[0013] Two raw scores are generated based on the importance weights, and the predicted probability of the target object is determined based on the two raw scores; wherein, pooling is used to average the first feature map for each feature channel;

[0014] If the predicted probability is less than or equal to the first threshold, it is determined that there is no target object in the image to be processed.

[0015] If the predicted probability is greater than the first threshold, it is determined that there is a target object in the image to be processed.

[0016] In one possible implementation, the method further includes:

[0017] Obtain the computing power resources of electronic devices based on their model or hardware configuration information;

[0018] If the computing power resources are lower than a first set threshold, or if the detection level of the target object in the detection request is lower than a second set threshold, the first threshold is increased by a set step value to improve the detection efficiency of the target object.

[0019] In one possible implementation, the pooling process is used to average the first feature map for each feature channel, including:

[0020] The average value is calculated for each feature channel corresponding to the first feature map. Based on the average value, each channel is compressed into a single scalar value. The single scalar value is then compared with the first threshold to determine whether a target object exists.

[0021] In one possible implementation, the step of using a second model to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object includes:

[0022] The image to be processed is input into the second model for feature extraction to obtain multiple second feature maps;

[0023] Obtain anchor points of different scales predefined at each spatial location for the first feature map or each second feature map, wherein the receptive field size of each second feature map is different;

[0024] Based on each anchor box, the second model is used to predict the classification score for each category and the offset of the bounding box, which is used to finely adjust the position and size of the anchor box to match the target object;

[0025] The preset anchor point frame is matched with the image frame of the target object in the image to be processed to determine the target object in the image to be processed.

[0026] According to a second aspect of this application, an image processing model applied to the aforementioned image processing method is provided, comprising:

[0027] A first model is used to extract a first feature map corresponding to the image to be processed in response to obtaining the image to be processed;

[0028] The second model is used to perform image recognition on the image to be processed and / or the feature map when a target object exists in the first feature map, so as to output a recognition result containing the target object;

[0029] Specifically, feature values ​​of the training images are extracted, the feature values ​​are normalized, and the normalized feature values ​​are used to train the first model to obtain the first model.

[0030] Semantic information of the image to be processed is obtained, and high-resolution spatial information and semantic information of shallow feature maps of the image to be processed are fused together and input into the training model for training. A feature pyramid with semantics and resolution adaptation is constructed to obtain the second model.

[0031] According to a third aspect of this application, an image processing apparatus is provided, the apparatus comprising:

[0032] An extraction unit is configured to extract a first feature map corresponding to the image to be processed using a first model in response to obtaining an image to be processed.

[0033] The recognition unit is configured to perform image recognition on the image to be processed and / or the feature map using a second model when a target object exists in the first feature map, so as to output a recognition result containing the target object;

[0034] The first model and the second model are different sub-models of the same model.

[0035] According to a fourth aspect of this application, an electronic device is provided, comprising:

[0036] At least one processor; and

[0037] A memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform:

[0039] In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model;

[0040] If a target object is present in the first feature map, the second model is used to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object;

[0041] The first model and the second model are different sub-models of the same model.

[0042] According to a fifth aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following: in response to obtaining an image to be processed, extracting a first feature map corresponding to the image to be processed using a first model; if a target object exists in the first feature map, performing image recognition on the image to be processed and / or the feature map using a second model to output a recognition result containing the target object; wherein the first model and the second model are different sub-models of the same model.

[0043] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0044] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0045] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0046] Figure 1 This paper illustrates the implementation flow of the image processing method according to an embodiment of the present application. Figure 1 ;

[0047] Figure 2 This paper illustrates the implementation flow of the image processing method according to an embodiment of the present application. Figure 2 ;

[0048] Figure 3 A schematic diagram of the image processing network model according to an embodiment of this application is shown;

[0049] Figure 4 A schematic diagram of the composition structure of the image processing apparatus according to an embodiment of this application is shown;

[0050] Figure 5 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0051] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] With the development of deep learning technology, object detection models based on Convolutional Neural Networks (CNNs) have been widely used in fields such as security, autonomous driving, and medical image analysis. These models are generally end-to-end single complex models. However, such single complex models have significant drawbacks: when processing each image, regardless of whether the image contains the target object of interest, the model needs to perform a complete, computationally complex forward inference process, resulting in a huge waste of computing power, especially in application scenarios where the target object appears infrequently, such as surveillance intrusion detection. In addition, for edge devices with limited computing power, such as mobile phones and embedded cameras, running a complete complex model will lead to high latency and high energy consumption, making it difficult to meet real-time requirements.

[0053] In response to the aforementioned technical problems, this application proposes the following technical solutions.

[0054] Figure 1 A schematic diagram illustrating the implementation flow of the image processing method according to an embodiment of this application is shown, as follows: Figure 1 As shown, the image processing method of this application embodiment includes the following processing steps:

[0055] Step 101: In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model.

[0056] Specifically, the image to be processed can come from a video stream captured in real time by a camera, or it can be an image file stored on an electronic device. The first model is a lightweight feature extraction and preliminary judgment sub-model. It can consist of a convolutional neural network (CNN) backbone and a detection and classification head.

[0057] Extracting the first feature map using the first model can be achieved by: inputting the image to be processed into the CNN backbone network of the first model, for example, using a lightweight network structure such as MobileNet or ShuffleNet. This network extracts the basic image features through multi-layer convolution and pooling operations. These features can represent low- to mid-level semantic information such as pixel grayscale, color features, contrast, edge features, and texture features. Based on these basic image features, a first feature map with multiple channels is generated.

[0058] As an example, the first model uses shallow networks (such as the first 3 layers of a CNN) to extract basic features of the image to be processed, and uses a lightweight classification head (such as global average pooling + fully connected layers) to quickly determine whether the input image contains the target object.

[0059] Determining whether a target object exists in the first feature map can be accomplished using the detection and classification head of the first model. Specifically, pooling is performed on the first feature map. Global average pooling (GAP) is used to calculate the average of the feature values ​​(Width × Height) of all spatial locations in the first feature map for each channel individually. The two-dimensional feature map of each channel is compressed into a single scalar value.

[0060] The scalar value sequence obtained from the above steps is input into a fully connected layer or processed directly to obtain the importance weights of each feature channel for the two categories of having a target object and not having a target object.

[0061] In this embodiment of the application, the first feature map is pooled by the detection classification head of the first model to obtain the importance weights of the feature channels of the first feature map; two original scores are generated based on the importance weights, and the prediction probability of the target object is determined based on the two original scores; wherein, the pooling process is used to average the first feature map for each feature channel.

[0062] If the predicted probability is less than or equal to the first threshold, it is determined that there is no target object in the image to be processed; if the predicted probability is greater than the first threshold, it is determined that there is a target object in the image to be processed.

[0063] Specifically, based on the importance weights of feature channels, two raw scores (e.g., logits) are generated, and then normalized using the Softmax function to a predicted probability P representing the presence of a target object. The predicted probability is compared with a first threshold, T, for example, T = 0.5. If the predicted probability P ≤ T, it is determined that there is no target object in the image to be processed, and the process ends or the next image is processed. If P > T, it is determined that a target object may exist in the image to be processed.

[0064] Step 102: If the target object exists in the first feature map, use the second model to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object.

[0065] In this embodiment of the application, the first model and the second model are different sub-models of the same model.

[0066] Since the first and second models are different parts of the same complete model, they can share low-level features. In a preferred embodiment, the first feature map can be directly input into the second model as its input, thus avoiding redundant computation. Alternatively, the original image to be processed can be re-inputted into the second model, which will then extract image features from the image to perform detailed recognition of the image containing the target object.

[0067] For example, the same model used for both the first and second models could be a model based on a Faster R-CNN or SSD (Single Shot MultiBox Detector) architecture. The first model corresponds to its Region Proposal Network (RPN) or a portion of the backbone network with a simple classifier, while the second model corresponds to the subsequent accurate classification and regression networks. This integrated training approach ensures consistency in feature understanding and decision-making between the two sub-models.

[0068] Taking the first model as the first 3 layers of a CNN as an example, the embodiments of this application can be implemented in the following ways:

[0069] Loading the image to be processed: Read the RGB format image to be processed, adjust the size of the image to be processed to the input size required by the pre-trained model (usually 224×224), and perform numerical normalization on the adjusted image to be processed: scale the pixel values ​​from [0,255] to [0,1]. Apply ImageNet statistics: subtract the mean [0.485,0.456,0.406] and divide by the standard deviation [0.229,0.224,0.225].

[0070] Convert the normalized image data to PyTorch tensor format and add batch dimension: [1,3,224,224].

[0071] The Conv1 layer processing of a CNN includes the following steps:

[0072] Convolution operation: A large convolution kernel (e.g., 7×7) is used for preliminary feature detection. The detected features include edge features, color features, and basic texture features. Output feature map: [1,64,112,112].

[0073] The Conv2 layer of a CNN, which includes groups of residual blocks or multiple stacked 3×3 convolutions, involves the following steps:

[0074] Feature Combination: Combines simple features of Conv1 into complex patterns; image features include shape contour features, texture features, simple object parts, etc., and the output feature map is: [1,128,56,56] or [1,256,28,28].

[0075] The Conv3 layer processing in CNN includes:

[0076] Image features are combined to form semantic features, the feature types of which include: object parts, complex structures, and meaningful patterns. The output feature map is [1,256,28,28] or [1,512,14,14]; it has strong semantic information, but the spatial resolution is reduced.

[0077] The global average pooling processing mechanism includes: spatial compression: calculating the average value of the two-dimensional feature map for each feature channel; input: [1,512,14,14], output: [1,512], compressing each 14×14 feature map into a single scalar value, thereby significantly reducing the number of processing parameters (from 100,352 dimensions to 512 dimensions).

[0078] A fully connected classification layer takes a 512-dimensional feature vector as input and a weight matrix of size [512, 2]. It learns the importance weights of each feature channel for both classes, where positive weights represent features containing the target object and negative weights represent features not containing the target object. The fully connected classification layer generates two raw scores: `score_positive` (strength of evidence supporting the inclusion of the target object) and `score_negative` (strength of evidence supporting the exclusion of the target object). If `score_positive > 0.5`, the prediction is that the target object is included; if `score_positive <= 0.5`, the prediction is that the target object is excluded.

[0079] Figure 2 A schematic diagram illustrating the implementation flow of the image processing method according to an embodiment of this application is shown, as follows: Figure 2 As shown, the image processing method of this application embodiment includes the following processing steps:

[0080] Step 201: In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model.

[0081] The processing steps in step 201 are exactly the same as those in step 101 mentioned above, and their implementation details will not be repeated here.

[0082] Step 202: Obtain the computing power resources of the electronic device based on the model or hardware configuration information of the electronic device; if the computing power resources are lower than the first set threshold, or the detection level of the target object in the detection request is lower than the second set threshold, increase the first threshold by a set step value to improve the detection efficiency of the target object.

[0083] To further enhance the adaptability of the method, embodiments of this application also include an adaptive adjustment step: the system acquires the model of the current electronic device or real-time hardware configuration information (such as CPU / GPU load, remaining memory, etc.) to assess its computing resources. When the computing resources are lower than a first set threshold (e.g., CPU utilization > 80%), or when the detection level specified in the received detection request is fast or low precision (lower than a second set threshold), the system automatically increases the first threshold T by a set step value (e.g., from 0.5 to 0.7). This makes the judgment conditions more stringent, ensuring that only images with higher confidence levels proceed to subsequent detailed inspection, thereby prioritizing processing speed and system smoothness when resources are scarce.

[0084] Step 203: If a target object exists in the first feature map, use the second model to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object.

[0085] The processing steps in step 203 are exactly the same as those in step 102 mentioned above, and their implementation details will not be repeated here.

[0086] In this embodiment, the image to be processed can be input into a second model for feature extraction to obtain multiple second feature maps; anchor boxes of different scales predefined at each spatial location can be obtained from the first feature map or each second feature map, with each second feature map having a different receptive field size; based on each anchor box, the second model is used to predict the classification score of each category and the offset of the bounding box, the offset being used to finely adjust the position and size of the anchor box to match the target object; the preset anchor boxes are matched with the image boxes of the target object in the image to be processed to determine the target object in the image to be processed.

[0087] Specifically, the second model in this application embodiment is a more complex detection sub-model, whose recognition process specifically includes: performing deep feature extraction to obtain multiple second feature maps. The second model typically contains deeper convolutional layers, extracting deeper, more semantically informative features based on the first feature map or the image to be processed, and outputting second feature maps at multiple scales, for example, from feature layers of different depths in the network. These feature maps have different receptive fields, each suitable for detecting targets of different sizes.

[0088] At each spatial location of each second feature map, a series of anchor boxes with different scales and aspect ratios are predefined. For each anchor box, the second model performs two prediction tasks:

[0089] Classification task: Predict the score of each target category (such as person, car, cat, etc.) contained within the anchor box.

[0090] Regression task: Predict the offset of the anchor box relative to its matched true target box in terms of center coordinates, width, and height. Use this offset to fine-tune the position and size of the preset anchor box to more accurately frame the target object.

[0091] Finally, post-processing algorithms such as Non-Maximum Suppression (NMS) are used to filter out a large number of overlapping and low-scoring adjusted anchor boxes, ultimately determining and outputting the positions (bounding boxes) and categories (recognition results) of all target objects in the image to be processed. It is important to emphasize that the first and second models are trained end-to-end as a single, unified model.

[0092] Since the first feature map may lack sufficient semantic information to detect targets at different scales, the FPN structure generates a series of feature maps with rich semantic information and different scales through top-down and lateral connections, thereby effectively improving the detection capability for targets of different sizes. Based on the multi-scale feature maps output by the FPN, two types of tasks are performed on each preset anchor box: one is a classification task, predicting the probability that each box contains targets of each category; the other is a regression task, predicting the position and size offset of each box relative to the ground truth target box. The output of the FPN detection head is post-processed (e.g., non-maximum suppression, NMS) to finally obtain the precise bounding boxes of all target objects in the image, along with their corresponding class labels and confidence scores, completing the entire processing flow.

[0093] In this embodiment, the multi-stage inference model employs an end-to-end joint training approach. During the training phase, a dataset containing images and their fully annotated data (whether there is a target, and the target's precise location and category) is input into the network. The total loss function L_total is designed as a weighted sum of the classification loss L_cls and the detection loss L_det:

[0094] L_total=α*L_cls+β*L_det

[0095] In this model, L_cls typically employs cross-entropy loss, while L_det usually combines Smooth L1 loss (for bounding box regression) and cross-entropy loss (for class classification). This joint training allows the classification and detection modules to optimize collaboratively, enabling the classification module to learn more accurately whether to activate the detection module, thus achieving a balance between accuracy and efficiency overall.

[0096] In resource-constrained scenarios or those requiring rapid response, the classification threshold θ can be increased by a step size Δ (e.g., from 0.4 to 0.6). This makes the classification module's judgment criteria more stringent; only images with higher confidence containing the target will enter the detection stage, thereby further reducing the frequency of calls to the detection module and improving overall processing speed. However, this may slightly increase the risk of missed detections.

[0097] In scenarios with ample resources or requiring high detection rates, the classification threshold θ can be reduced by a step size Δ, for example, from 0.4 to 0.3. This makes the classification module more lenient in its judgment, allowing more suspected target images to enter the detection stage for confirmation, which helps reduce the false negative rate and improve system accuracy, but will correspondingly increase the computational load.

[0098] Through this dynamic adaptive mechanism, the embodiments of this application can intelligently adapt to different operating environments and task requirements, achieving the best trade-off between accuracy and speed.

[0099] Figure 3 A schematic diagram of the image processing network model according to an embodiment of this application is shown, as follows: Figure 3 As shown, the image processing network model of this application embodiment includes an image processing detection network, which includes a shallow network and a deep network. After processing by the shallow network, the generated first feature map is input into the classification network. The classification network judges the target object and determines whether to input the first feature map into the deep network based on the judgment result.

[0100] This application's embodiments utilize shallow networks (such as the first 3 layers of a CNN) to extract basic features, and use a lightweight classification head (such as global average pooling + fully connected layers) to quickly determine whether the input image contains a target. Only for the feature maps of the images to be processed with the classification result of "containing target object", the shallow features are reused and subsequent deep networks (such as FPN + detection head) are activated to complete target localization. The shallow network consists of two main modules: a classification module: inputting the image to be processed (RGB format); and a feature extraction module: using the first 3 layers (Conv1~Conv3) of a pre-trained model and outputting feature maps; the classification head is executed using global average pooling and fully connected layers (outputting binary classification probabilities).

[0101] Deep networks can directly receive Conv3 output features from the classification module. The classification network is a detection network, which can incorporate an FPN structure to fuse multi-scale features and output target bounding boxes and confidence scores. Images without targets are filtered out during the classification stage, skipping invalid detection calculations. A multi-task joint training framework is adopted to uniformly optimize the classification loss (cross-entropy) and detection loss (Smooth L1), avoiding error accumulation caused by cascaded structures.

[0102] The image processing network of this application embodiment is compatible with mainstream detection frameworks (such as SSD and RetinaNet) and does not require modification of the model backbone; the technical solution of this application embodiment can be extended to multi-task scenarios, such as simultaneous detection of faces and license plates.

[0103] This application embodiment also describes an image processing model applied to the aforementioned image processing method, comprising: a first model, configured to extract a first feature map corresponding to the image to be processed in response to obtaining an image to be processed; a second model, configured to perform image recognition on the image to be processed and / or the feature map when a target object exists in the first feature map, so as to output a recognition result containing the target object; wherein, feature values ​​of a training image are extracted, the feature values ​​are normalized, and the normalized feature values ​​are used to train the first model to obtain the first model; semantic information of the image to be processed is obtained, and high-resolution spatial information and semantic information of the shallow feature map of the image to be processed are fused and input into the training model for training to construct a feature pyramid with semantics and resolution adaptation to obtain the second model.

[0104] The training process for the first and second models will not be elaborated here.

[0105] This application embodiment splits a single model into a first sub-model and a second sub-model that share basic features. The more computationally intensive second sub-model is activated only when the first sub-model determines that a target may exist. This avoids unnecessary complex calculations on images without target objects, significantly reducing average processing time. Since the first and second models are different sub-models of the same model, they share most of the underlying feature extraction network, ensuring feature consistency. The first model's judgment is based on high-quality feature maps, resulting in a low false positive rate and guaranteeing final detection accuracy. A dynamic threshold adjustment mechanism based on device computing power or detection level is introduced. In scenarios with limited computing power or low requirements, increasing the judgment threshold can filter out more uncertainties, further reducing the frequency of calling the second model and improving the system's adaptability and practicality. This application embodiment is particularly suitable for deployment on edge computing devices, effectively alleviating the computational pressure on electronic devices, meeting real-time processing needs, and broadening the application boundaries of complex image recognition models.

[0106] Figure 4 A schematic diagram of the composition structure of the image processing apparatus according to an embodiment of this application is shown, such as... Figure 4 As shown, the image processing apparatus of this application embodiment includes:

[0107] Extraction unit 40 is used to extract a first feature map corresponding to the image to be processed using a first model in response to obtaining the image to be processed;

[0108] The recognition unit 41 is used to perform image recognition on the image to be processed and / or the feature map using a second model when a target object exists in the first feature map, so as to output a recognition result containing the target object;

[0109] The first model and the second model are different sub-models of the same model.

[0110] In this embodiment of the application, the first model includes a convolutional neural network. The extraction of a first feature map corresponding to the image to be processed using the first model includes: inputting the image to be processed into the first model, extracting the basic image features of the image to be processed through the convolutional neural network of the first model, and generating a first feature map based on the basic image features. The basic image features represent at least one of pixel grayscale, color features, contrast, edge features, and texture features.

[0111] The first feature map is pooled using the detection and classification head of the first model to obtain the importance weights of the feature channels. Two raw scores are generated based on these importance weights, and the predicted probability of the target object is determined based on these two raw scores. Specifically, the pooling process is used to average the first feature map for each feature channel. The average value is calculated for each feature channel corresponding to the first feature map, and each channel is compressed into a single scalar value based on the average value. This single scalar value is then compared with a first threshold to determine whether a target object exists. If the predicted probability is less than or equal to the first threshold, it is determined that there is no target object in the image to be processed; if the predicted probability is greater than the first threshold, it is determined that a target object exists in the image to be processed.

[0112] The computing power resources of the electronic device are obtained based on the model or hardware configuration information of the electronic device; if the computing power resources are lower than the first set threshold, or if the detection level of the target object in the detection request is lower than the second set threshold, the first threshold is increased by a set step value to improve the detection efficiency of the target object.

[0113] The image to be processed is input into the second model for feature extraction, resulting in multiple second feature maps. Anchor boxes of different scales are obtained at each spatial location from the first feature map or each second feature map, with each second feature map having a different receptive field size. Based on each anchor box, the second model is used to predict the classification score for each category and the offset of the bounding box. The offset is used to finely adjust the position and size of the anchor box to match the target object. The preset anchor boxes are matched with the image boxes of the target object in the image to be processed to determine the target object in the image to be processed.

[0114] In an exemplary embodiment, each processing unit in the image processing apparatus of this application embodiment may be implemented by one or more central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, micro controller units (MCUs), microprocessors, or other electronic components.

[0115] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0116] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0117] Figure 5 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0118] like Figure 5As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0127] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0129] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, the method comprising: In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model; If a target object is present in the first feature map, the second model is used to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object; The first model and the second model are different sub-models of the same model.

2. The method according to claim 1, wherein the first model comprises a convolutional neural network, and the step of extracting a first feature map corresponding to the image to be processed using the first model comprises: The image to be processed is input into the first model, and the basic image features of the image to be processed are extracted through the convolutional neural network of the first model; The first feature map is generated based on the image's basic features; the image's basic features represent at least one of pixel grayscale, color features, contrast, edge features, and texture features.

3. The method according to claim 1, further comprising: The first feature map is pooled using the detection and classification head of the first model to obtain the importance weights of the feature channels of the first feature map. Two raw scores are generated based on the importance weights, and the predicted probability of the target object is determined based on the two raw scores; wherein, pooling is used to average the first feature map for each feature channel; If the predicted probability is less than or equal to the first threshold, it is determined that there is no target object in the image to be processed. If the predicted probability is greater than the first threshold, it is determined that there is a target object in the image to be processed.

4. The method according to claim 3, further comprising: Obtain the computing power resources of electronic devices based on their model or hardware configuration information; If the computing power resources are lower than a first set threshold, or if the detection level of the target object in the detection request is lower than a second set threshold, the first threshold is increased by a set step value to improve the detection efficiency of the target object.

5. The method according to claim 3, wherein the pooling process is used to average the first feature map for each feature channel, comprising: The average value is calculated for each feature channel corresponding to the first feature map. Based on the average value, each channel is compressed into a single scalar value. The single scalar value is then compared with the first threshold to determine whether a target object exists.

6. The method according to claim 1, wherein the step of performing image recognition on the image to be processed and / or the feature map using the second model to output a recognition result containing the target object includes: The image to be processed is input into the second model for feature extraction to obtain multiple second feature maps; Obtain anchor points of different scales predefined at each spatial location for the first feature map or each second feature map, wherein the receptive field size of each second feature map is different; Based on each anchor box, the second model is used to predict the classification score for each category and the offset of the bounding box, which is used to finely adjust the position and size of the anchor box to match the target object; The preset anchor point frame is matched with the image frame of the target object in the image to be processed to determine the target object in the image to be processed.

7. An image processing model applied to the method of any one of claims 1 to 6, comprising: A first model is used to extract a first feature map corresponding to the image to be processed in response to obtaining the image to be processed; The second model is used to perform image recognition on the image to be processed and / or the feature map when a target object exists in the first feature map, so as to output a recognition result containing the target object; Specifically, feature values ​​of the training images are extracted, the feature values ​​are normalized, and the normalized feature values ​​are used to train the first model to obtain the first model. Semantic information of the image to be processed is obtained, and high-resolution spatial information and semantic information of shallow feature maps of the image to be processed are fused together and input into the training model for training. A feature pyramid with semantics and resolution adaptation is constructed to obtain the second model.

8. An image processing apparatus, the apparatus comprising: An extraction unit is configured to extract a first feature map corresponding to the image to be processed using a first model in response to obtaining an image to be processed. The recognition unit is configured to perform image recognition on the image to be processed and / or the feature map using a second model when a target object exists in the first feature map, so as to output a recognition result containing the target object; The first model and the second model are different sub-models of the same model.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform: In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model; If a target object is present in the first feature map, the second model is used to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object; The first model and the second model are different sub-models of the same model.

10. A storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing: In response to obtaining the image to be processed, a first feature map corresponding to the image to be processed is extracted using a first model; If a target object is present in the first feature map, the second model is used to perform image recognition on the image to be processed and / or the feature map to output a recognition result containing the target object; in, The first model and the second model are different sub-models of the same model.

Images