Image processing method and image processing apparatus

Abstract

An image processing method and an image processing apparatus are provided. The method includes processing an image using a target recognition model to obtain a target probability distribution map and type and quantity information of each type of target in a first type. The target probability distribution map includes the target probability corresponding to each pixel. The first type is determined based on the pixel value of the target in the image and a preset classification threshold. Based on a first preset processing method, image processing is performed on the target probability distribution map to obtain a target grayscale image and a target heatmap. A morphological closing operation is performed on the target grayscale image to obtain multiple connected regions. Each connected region represents a clustering region of a type of target in a second type, which is determined based on the pixel value of the target in the image and a preset classification threshold. For each connected region, a group recognition processing is performed using a second preset processing method to obtain the type and quantity information of targets within the connected region.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Animals are an essential component of natural ecosystems and a crucial indicator for biodiversity assessment and sustainable ecological development. Installing cameras in ecological reserves allows for animal counting and research without disturbing the animals. Advances in artificial intelligence technology have enabled computing devices to quickly and accurately detect animal distribution, species, and numbers from images, much like zoologists can.

[0003] In the process of identifying the species and number of animals, for some animals that like to live in dense groups, such as certain birds, the identification of each individual species is not only computationally intensive but also has low accuracy due to unclear image quality and mutual occlusion. Summary of the Invention

[0004] In view of the above, embodiments of this disclosure provide an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium.

[0005] One aspect of this disclosure provides an image processing method, including:

[0006] The image is processed using a target recognition model to obtain a target probability distribution map and the type and quantity information of each type of target in the first type. The target probability distribution map includes the target probability corresponding to each pixel. The first type is determined based on the pixel value of the target in the image and a preset classification threshold.

[0007] Based on the first preset processing method, the above target distribution probability map is processed to obtain a target grayscale map and a target heat map.

[0008] The above target grayscale image is subjected to an image morphological closing operation to obtain multiple connected regions, wherein each of the above connected regions represents a clustering region of a type of target in the second type, and the second type is determined based on the pixel value of the above target in the above image and the above preset classification threshold.

[0009] For each of the aforementioned connected regions, a second preset processing method is used to perform group identification processing on the aforementioned connected regions to obtain the type and quantity information of the aforementioned targets within the aforementioned connected regions.

[0010] According to embodiments of this disclosure, the image processing method further includes:

[0011] Based on the type and quantity information of each type of target in the first type mentioned above, and the type and quantity information of the targets in multiple connected areas, the target statistics information of the above image is generated.

[0012] Based on the above image, and according to the above target heat map and the above target statistical information, a display image is generated, wherein the above display image includes the marking information of multiple types of targets of the above first type;

[0013] The above images are displayed visually.

[0014] According to embodiments of this disclosure, the target distribution probability map includes a plurality of first pixel rows;

[0015] The above-mentioned image processing of the target distribution probability map based on the first preset processing method to obtain the target grayscale map and the target heatmap includes:

[0016] For each of the aforementioned first pixel rows, pixel-by-pixel integration is performed on the aforementioned first pixel rows to obtain the total number of targets corresponding to the aforementioned image;

[0017] After performing pixel-by-pixel integration, the above target probability map is normalized so that the pixel value of each pixel in the target probability map is within a first preset range.

[0018] The normalized target probability distribution map is subjected to a first mapping process to obtain the above target grayscale map, wherein the pixel value of each pixel in the above target grayscale map is within a second preset range.

[0019] The target grayscale image is processed by color space to obtain a pseudo-color image, which represents the target heat map.

[0020] According to embodiments of this disclosure, prior to performing the above-described pixel-by-pixel integration process, the method further includes:

[0021] Based on a preset pixel threshold, the above target distribution probability map is subjected to threshold filtering to obtain a filtered target distribution probability map, and then pixel-by-pixel integration is performed on the filtered target distribution probability map.

[0022] The above-mentioned target statistics include the total number of the aforementioned targets.

[0023] According to embodiments of this disclosure, the target grayscale image includes multiple unconnected initial regions, wherein each initial region includes a plurality of clustered targets;

[0024] The aforementioned image morphological closing operation on the target grayscale image yields multiple connected regions, including:

[0025] For any two initial regions mentioned above, if the distance between the two initial regions meets the distance threshold, the structuring element in the image morphology closing operation is used to dilate the two initial regions to obtain a transition region, wherein the two dilated initial regions in the transition region partially overlap.

[0026] The transition region is eroded using the structuring element in the above image morphological closing operation to obtain a connected region. The difference between the image size of the connected region and the image size of the initial region satisfies a preset classification threshold, and the two eroded initial regions in the connected region partially overlap.

[0027] According to embodiments of this disclosure, the above-described image processing using a target recognition model to obtain a target distribution probability map includes:

[0028] The image is acquired by an image acquisition device, wherein the resolution of the image is a preset resolution and the image includes at least one shadow area.

[0029] The target recognition model is used to identify the target type in the image to obtain the target probability distribution map;

[0030] After obtaining the target distribution probability map, the process also includes:

[0031] For each of the aforementioned shadow regions, if the pixel length or pixel width of the shadow region is greater than a preset classification threshold, the shadow region is deleted from the image to obtain a filtered image, and the target distribution probability map is generated based on the filtered image.

[0032] According to embodiments of this disclosure, the filtered image includes a target region that meets the preset classification threshold and other regions that do not meet the preset classification threshold. The target region includes at least one type of target of the first type. The target recognition model includes a target detection algorithm and a density estimation algorithm.

[0033] The above-mentioned target recognition model is used to identify the target type of the filtered image to obtain the target distribution probability map, including:

[0034] For the aforementioned target area, the aforementioned target detection algorithm is used to perform type identification processing on the targets in the aforementioned target area to obtain the type information and quantity information of the aforementioned first type corresponding to the aforementioned target area;

[0035] The density estimation algorithm described above is used to process each pixel in the target region and other regions to obtain the target probability corresponding to each pixel.

[0036] According to embodiments of this disclosure, the above-mentioned identification processing of the connected area using the second preset processing method to obtain the type and quantity information of the targets within the connected area includes:

[0037] The above-mentioned connected regions are processed using a preset classification model to obtain the above-mentioned category information of the above-mentioned targets within the above-mentioned connected regions, wherein the above-mentioned preset classification model is constructed based on a convolutional neural network;

[0038] For each second pixel row in the aforementioned connected region, pixel-by-pixel integration is performed on the aforementioned second pixel row to obtain the aforementioned quantity information of the target corresponding to the aforementioned connected region.

[0039] According to an embodiment of this disclosure, the target recognition model is trained using a first training image and first label data, wherein the first training image includes a plurality of first training targets, and the first label data characterizes the position and type of the plurality of first training targets in the first training image.

[0040] The aforementioned preset classification model is trained using the second training image and the second label data, wherein the second training image includes the second training target, and the second label data represents the type of the second training target.

[0041] Another aspect of this disclosure provides an image processing apparatus, comprising:

[0042] The first processing module is used to process the image using a target recognition model to obtain a target probability distribution map and the type and quantity information of each type of target in the first type. The target probability distribution map includes the target probability corresponding to each pixel, and the first type is determined based on the pixel value of the target in the image and a preset classification threshold.

[0043] The second processing module is used to perform image processing on the above target distribution probability map based on the first preset processing method to obtain a target grayscale map and a target heat map.

[0044] The third processing module is used to perform image morphological closing operation on the above target grayscale image to obtain multiple connected regions, wherein each of the above connected regions represents a clustering region of a type of target in the second type, and the second type is determined based on the pixel value of the above target in the above image and the above preset classification threshold.

[0045] The identification module is used to perform group identification processing on each of the above-mentioned connected areas using a second preset processing method to obtain the type and quantity information of the above-mentioned targets within the connected areas.

[0046] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0047] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0048] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0049] According to embodiments of this disclosure, individual targets of the first type are detected using a target recognition model. Then, the target distribution probability map generated by the target recognition model is processed using a first preset processing method to obtain a target grayscale map and a target heatmap. Image morphological closing operations are performed on the target grayscale map to generate multiple connected regions. Group recognition processing is performed on the connected regions using a second preset processing method to obtain the type and quantity information of the targets in each connected region. Since image morphological closing operations can integrate unconnected regions, the accuracy of group recognition is improved. Thus, accurate identification of the type and quantity of targets in the image is achieved through the fusion of individual recognition and group recognition. Attached Figure Description

[0050] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0051] Figure 1 An exemplary system architecture to which image processing methods can be applied according to embodiments of the present disclosure is illustrated;

[0052] Figure 2 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown schematically.

[0053] Figure 3 The illustration is a schematic diagram of a display image according to an embodiment of the present disclosure;

[0054] Figure 4 A flowchart illustrating an image morphological closing operation according to an embodiment of the present disclosure is shown schematically.

[0055] Figure 5 A schematic diagram of a structural element according to an embodiment of the present disclosure is shown;

[0056] Figure 6A block diagram schematically illustrates an image processing apparatus according to embodiments of the present disclosure; and

[0057] Figure 7 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Detailed Implementation

[0058] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0060] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0061] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0062] Embodiments of this disclosure provide an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. The method includes processing an image using a target recognition model to obtain a target probability distribution map and type and quantity information of each type of target in a first type. The target probability distribution map includes the target probability corresponding to each pixel, and the first type is determined based on the pixel values ​​of the target in the image and a preset classification threshold. Based on a first preset processing method, image processing is performed on the target probability distribution map to obtain a target grayscale image and a target heatmap. A morphological closing operation is performed on the target grayscale image to obtain multiple connected regions, where each connected region represents a clustering region of a type of target in a second type, and the second type is determined based on the pixel values ​​of the target in the image and a preset classification threshold. For each connected region, a group recognition processing is performed on the connected region using a second preset processing method to obtain the type and quantity information of targets within the connected region.

[0063] Figure 1 An exemplary system architecture 100 to which image processing methods can be applied according to embodiments of this disclosure is illustrated schematically. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0064] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, a server 105, and an image acquisition device 106. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103, the server 105, and the image acquisition device 106. The network 104 may include various connection types, such as wired and / or wireless communication links.

[0065] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0066] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0067] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0068] The image acquisition device 106 can capture images of the external environment to obtain images of multiple targets, which can be used to determine the type and number of targets.

[0069] It should be noted that the image processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the image processing apparatus provided in this embodiment can generally be located in server 105. The image processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the image processing apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the image processing method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Correspondingly, the image processing apparatus provided in this embodiment can also be located in terminal devices 101, 102, or 103, or in other terminal devices different from terminal devices 101, 102, or 103.

[0070] It should be understood that Figure 1 The number of terminal devices, networks, servers, and image acquisition devices shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, servers, and image acquisition devices can be included.

[0071] Figure 2 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown schematically.

[0072] like Figure 2 As shown, the image processing method includes operations S201 to S204.

[0073] In operation S201, the image is processed using a target recognition model to obtain a target probability distribution map and the type and quantity information of each type of target in the first type. The target probability distribution map includes the target probability corresponding to each pixel. The first type is determined based on the pixel value of the target in the image and a preset classification threshold.

[0074] In operation S202, based on the first preset processing method, image processing is performed on the target distribution probability map to obtain the target grayscale map and the target heat map.

[0075] In operation S203, an image morphological closing operation is performed on the target grayscale image to obtain multiple connected regions. Each connected region represents a clustering region of a type of target in the second type, which is determined based on the pixel value of the target in the image and a preset classification threshold.

[0076] In operation S204, for each connected area, the second preset processing method is used to perform group identification processing on the connected area to obtain the type and quantity information of the targets in the connected area.

[0077] According to the embodiments of this disclosure, in order to facilitate the description of this disclosure, the following embodiments are illustrated with birds as the target. It should be noted that the methods of this disclosure are not only applicable to birds, but also applicable to other animals or targets, such as hyenas, monkey troops, vehicles on the road, etc.

[0078] According to embodiments of this disclosure, the preset classification threshold is specifically set based on the image resolution, and is used to separate birds of the first type that are larger in size. For example, at the preset classification threshold... When the value is 50, the length and width of the pixels occupied by the target in the image are compared with the preset classification threshold. If either the length or the width is greater than the preset classification threshold, the target is identified as a first type of target; otherwise, it is identified as a second type of target.

[0079] According to embodiments of this disclosure, each pixel in the target probability map represents the probability that the location is an animal. A connected region can refer to an area in an image where multiple birds of the same type are clustered, with overlapping areas between different animals within the connected region. The target heatmap visually displays the density of animals using different colors; for example, the denser the animals, the darker the color, and the sparser the animals, the lighter the color.

[0080] According to an embodiment of this disclosure, after acquiring an image, a first type of animal is first identified using a preset classification threshold to obtain a target distribution probability map that includes species information and quantity information of different types of animals in the first type. In the target distribution probability map, each pixel in the other regions except the first type region corresponds to a probability that the location is an animal. The species information can refer to species information, such as white crane, goose, red-crowned crane, sparrow, etc.

[0081] According to embodiments of this disclosure, a target probability distribution map is processed using a first preset processing method, such as performing integration and mapping processing on the target probability distribution map, to obtain a target grayscale map and a target heatmap. Since animals of the same species tend to cluster together, gaps may exist in the target grayscale map due to multiple animals of the same species not being completely clustered together. Therefore, performing an image morphological closing operation on the target grayscale map can merge two clustered regions with smaller gaps into a connected region. Thus, a second preset processing method can be used to perform group identification on each connected region, obtaining species and quantity information within each connected region.

[0082] According to embodiments of this disclosure, individual targets of the first type are detected using a target recognition model. Then, the target distribution probability map generated by the target recognition model is processed using a first preset processing method to obtain a target grayscale map and a target heatmap. Image morphological closing operations are performed on the target grayscale map to generate multiple connected regions. Group recognition processing is performed on the connected regions using a second preset processing method to obtain the type and quantity information of targets in each connected region. Since image morphological closing operations can integrate unconnected regions, the accuracy of group recognition is improved. Thus, accurate identification of the type and quantity of targets in the image is achieved through the fusion of individual recognition and group recognition.

[0083] Figure 3 A schematic diagram of a display image according to an embodiment of the present disclosure is shown.

[0084] According to embodiments of this disclosure, the image processing method further includes the following operations:

[0085] Based on the type and quantity information of each type of target in the first type and the type and quantity information of targets in multiple connected areas, target statistics information for the image is generated; based on the image, a display image is generated according to the target heat map and target statistics information, wherein the display image includes the marking information of multiple types of targets in the first type; the display image is then visualized.

[0086] According to embodiments of this disclosure, after determining the number of each type and species of animal in the image, target statistics are generated based on the species and quantity information of each species of target in the first type and the species and quantity information of targets in multiple connected regions. The target statistics and target heatmap are then combined on the image to generate the final display image, such as... Figure 3 As shown, the displayed image is finally visualized using a display device.

[0087] In one exemplary embodiment, when generating the display image, the original image can be placed in the center of the visualization interface, occupying the entire screen. Several large, sparsely distributed animals of the first type are marked in the upper left corner with an elliptical outline, while the large central area represents clustered animals. A target heatmap can be displayed in the lower right corner of the original image. The target heatmap's size is set to w*h, for example, 1 / 5 of the original image. The background in the target heatmap is represented by blue, and the colors change from cyan to green, yellow, and red as the animal density increases. The target statistics in the lower left corner of the original image include the total number and species of animals in the image's field of view. These statistics include the major category, specific species, and the specific quantity of each species.

[0088] It should be noted that the display format of the above-mentioned images can be adjusted according to actual needs. For example, the target heat map in the lower right corner can be adjusted to any area of ​​the original image. The above examples are only for illustrative purposes and are not intended to limit the display images of this disclosure to only the above examples.

[0089] According to embodiments of this disclosure, the target probability distribution map includes a plurality of first pixel rows.

[0090] Specifically, based on the first preset processing method, image processing is performed on the target distribution probability map to obtain the target grayscale map and the target heat map, including the following operations:

[0091] For each first pixel row, perform pixel-by-pixel integration to obtain the total number of targets corresponding to the image and the integrated first pixel row.

[0092] After pixel-by-pixel integration, the target probability map is normalized so that the pixel value of each pixel in the target probability map is within a first preset range.

[0093] The normalized target probability distribution map is subjected to a first mapping process to obtain a target grayscale image, wherein the pixel value of each pixel in the target grayscale image is within a second preset range.

[0094] A second mapping process is performed on the target grayscale image using color space to obtain a pseudocolor image, which represents the target heat map.

[0095] According to the embodiments of this disclosure, since each pixel in the target probability distribution map represents the probability that the position is an animal, the number of animals can be obtained by integrating the pixel. Therefore, for each pixel row in the target probability distribution map, pixel-by-pixel integration can be performed to obtain the total number of animals corresponding to the image and the target probability distribution map. By normalizing the target probability distribution map, a normalized target probability distribution map with pixel values ​​normalized to a first preset range can be obtained. The first preset range can be the interval [0,1]. The formula used for normalization is shown in formula (1).

[0096]

[0097] in, Let x and y be any pixel, and x and y be the x and y coordinates of that pixel, respectively. This represents the maximum pixel value among all pixels. For example... When the value is 200, the pixel value of each pixel in the image needs to be divided by 200. This normalization process can normalize all pixel values ​​of the target probability map to the [0,1] interval.

[0098] According to embodiments of this disclosure, a first mapping process is performed on the normalized target probability distribution map to obtain a target grayscale image with pixel values ​​within a second preset range. For example, each pixel value is multiplied by 255 to map to the [0, 255] interval. The target grayscale image is then converted into a pseudo-color image using the HSV color space, wherein the configuration parameters of the HSV color space are: saturation and brightness are set to 1.0 and 1.0 respectively, and hue is mapped from grayscale 0 to 255 to 0° to 255°.

[0099] According to embodiments of this disclosure, pixel-by-pixel integration is performed on the first pixel row to obtain the total number of objects corresponding to the image, including the following operations:

[0100] The pixel values ​​in the first pixel row are summed to obtain the total pixel value of the first pixel row; the total pixel value is then integrated to obtain the total number of objects. The object statistics include the total number of objects.

[0101] According to embodiments of this disclosure, since image pixels are discrete values, pixel-by-pixel integration involves adding each image pixel together. The accumulated result may be a non-integer, so a rounding down is performed to obtain the target total.

[0102] According to embodiments of this disclosure, when generating and displaying an image, the total number of targets in the image can also be displayed in the target statistics information in the lower left corner. This total number can be the total number of targets of all types and all kinds, or the total number of targets of a certain type.

[0103] According to embodiments of this disclosure, before performing pixel-by-pixel integration processing, the following operations are also included:

[0104] Based on a preset pixel threshold, the target probability distribution map is subjected to threshold filtering to obtain a filtered target probability distribution map, which is then subjected to pixel-by-pixel integration.

[0105] According to embodiments of this disclosure, to avoid the influence of environmental factors or the system performance of the image acquisition device, each pixel can be filtered using a preset pixel threshold. For example, at a preset pixel threshold... In this case, pixels with values ​​less than T are set to 0, and the probability that the pixel is an animal is almost negligible.

[0106] Figure 4 A flowchart illustrating an image morphological closing operation according to an embodiment of the present disclosure is shown.

[0107] Figure 5 A schematic diagram of a structural element according to an embodiment of the present disclosure is shown.

[0108] According to embodiments of this disclosure, the target grayscale image includes a plurality of unconnected initial regions, wherein each initial region includes a plurality of clustered targets.

[0109] like Figure 4 As shown, image morphological closing operations are performed on the target grayscale image to obtain multiple connected regions, including the following operations:

[0110] For any two initial regions, if the distance between the two initial regions meets the distance threshold, the structuring element in the image morphology closing operation is used to dilate the two initial regions to obtain a transition region, in which the two dilated initial regions in the transition region partially overlap.

[0111] The transition region is eroded using structuring elements in the image morphology closing operation to obtain a connected region. The difference between the image size of the connected region and the image size of the initial region satisfies a preset classification threshold, and the two eroded initial regions in the connected region partially overlap.

[0112] According to embodiments of this disclosure, the distance threshold can be specifically set according to actual conditions, for example, it can be the length of n pixels. If the distance between two initial regions is less than this length, then image morphological closing operations such as... Figure 5 The structuring element shown expands two initial regions, causing the two expanded initial regions to partially overlap. See [link / reference]. Figure 4This results in a larger transition region. Erosion is then applied to this transition region. The areas that have already been connected will not be erased by erosion, but small scattered points within the transition region can still be removed. This restores the original enlarged area while removing small scattered points from the image.

[0113] According to embodiments of this disclosure, see Figure 5 During the expansion process, pixels can expand in four directions: up, down, left, and right, or expand simultaneously in other directions.

[0114] It should be noted that the method disclosed herein is based on the premise that animals of different species do not usually congregate together, and only animals of the same species tend to live in dense groups (such as some birds). Therefore, animals identified in the same connected area can be considered to be of the same species.

[0115] According to embodiments of this disclosure, an image is processed using a target recognition model to obtain a target probability distribution map, including the following operations:

[0116] An image is acquired by an image acquisition device, wherein the image resolution is a preset resolution and the image includes at least one shadow region. The target type in the image is identified using the target recognition model to obtain the target probability distribution map.

[0117] After obtaining the target distribution probability map, the method further includes: for each shadow region, if the pixel length or pixel width of the shadow region is greater than a preset classification threshold, deleting the shadow region in the image to obtain a filtered image, so as to generate the target distribution probability map based on the filtered image.

[0118] According to embodiments of this disclosure, the image acquisition device may include a camera, a video camera, or a webcam, etc.

[0119] In one exemplary embodiment, an image acquisition device is used to acquire images of an ecological reserve. The resolution of these images can be a preset 3840*2160, and a preset classification threshold is set. When the value is 50, the pixel length or pixel width of the shadow area on the image is detected by the preset classification threshold. After obtaining the target type and quantity in the shadow area, the shadow area is deleted to obtain the filtered image, and the target distribution probability map is generated based on the filtered image.

[0120] According to embodiments of this disclosure, the filtered image includes a target region that meets a preset classification threshold and other regions that do not meet the preset classification threshold. The target region includes at least one type of target of a first type. The target recognition model includes a target detection algorithm and a density estimation algorithm.

[0121] According to embodiments of this disclosure, a target recognition model is used to identify the target type of the filtered image to obtain a target distribution probability map, including the following operations:

[0122] For the target area, the target detection algorithm is used to perform type identification processing on the targets in the target area to obtain the type information and quantity information of the first type corresponding to the target area;

[0123] The density estimation algorithm is used to process each pixel in the target region and other regions to obtain the target probability corresponding to each pixel.

[0124] According to embodiments of this disclosure, the target region refers to a first type of animal that is relatively large and does not like to gather in groups. For the target region, a target detection algorithm is used to perform type identification processing on the targets in the target region, obtaining the species information and quantity information of the first type corresponding to the target region. The target detection algorithm can be a YOLO series target detection algorithm.

[0125] According to embodiments of this disclosure, other areas contain some second-type animals that like to gather, such as sparrows. A density estimation algorithm based on a convolutional neural network is used to process each pixel in the target area and other areas to obtain the target probability corresponding to each pixel, thereby obtaining a target probability distribution map.

[0126] According to embodiments of this disclosure, a second preset processing method is used to identify the connected area to obtain the type and quantity information of targets within the connected area, including the following operations:

[0127] The connected regions are processed using a pre-defined classification model to obtain the type information of targets within the connected regions. The pre-defined classification model is based on a convolutional neural network. For each second pixel row in the connected regions, pixel-by-pixel integration is performed on the second pixel row to obtain the quantity information of targets corresponding to the connected regions.

[0128] According to embodiments of this disclosure, the preset classification model can be a classification algorithm based on a convolutional neural network. This preset classification model can identify the type information of targets within a connected region, and then perform pixel-by-pixel integration on each second pixel row within the connected region to determine the quantity information of targets within each connected region. Pixel-by-pixel integration specifically involves accumulating the image pixel values.

[0129] According to an embodiment of the present disclosure, the target recognition model is trained using a first training image and first label data, wherein the first training image includes a plurality of first training targets, and the first label data characterizes the position and type of the plurality of first training targets in the first training image;

[0130] The preset classification model is trained using the second training image and the second label data. The second training image includes the second training target, and the second label data represents the type of the second training target.

[0131] Figure 6 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0132] like Figure 6 As shown, the image processing device 600 includes a first processing module 610, a second processing module 620, a third processing module 630, and a recognition module 640.

[0133] The first processing module 610 is used to process the image using a target recognition model to obtain a target probability distribution map and the type and quantity information of each type of target in the first type. The target probability distribution map includes the target probability corresponding to each pixel point, and the first type is determined based on the pixel value of the target in the image and a preset classification threshold.

[0134] The second processing module 620 is used to perform image processing on the target distribution probability map based on the first preset processing method to obtain the target grayscale map and the target heat map.

[0135] The third processing module 630 is used to perform image morphological closing operations on the target grayscale image to obtain multiple connected regions, wherein each connected region represents a clustering region of a type of target in the second type, and the second type is determined based on the pixel value of the target in the image and a preset classification threshold.

[0136] The identification module 640 is used to perform group identification processing on each connected area using a second preset processing method to obtain the type and quantity information of targets within the connected area.

[0137] According to embodiments of this disclosure, individual targets of the first type are detected using a target recognition model. Then, the target distribution probability map generated by the target recognition model is processed using a first preset processing method to obtain a target grayscale map and a target heatmap. Image morphological closing operations are performed on the target grayscale map to generate multiple connected regions. Group recognition processing is performed on the connected regions using a second preset processing method to obtain the type and quantity information of targets in each connected region. Since image morphological closing operations can integrate unconnected regions, the accuracy of group recognition is improved. Thus, accurate identification of the type and quantity of targets in the image is achieved through the fusion of individual recognition and group recognition.

[0138] According to embodiments of this disclosure, the image processing apparatus 600 further includes a first generation module, a second generation module, and a display module.

[0139] The first generation module is used to generate target statistics information of the image based on the type and quantity information of each type of target in the first type and the type and quantity information of targets in multiple connected regions.

[0140] The second generation module is used to generate a display image based on the image, according to the target heatmap and target statistics, wherein the display image includes the labeling information of multiple types of targets of the first type.

[0141] The display module is used to visually display images.

[0142] According to embodiments of this disclosure, the target probability distribution map includes a plurality of first pixel rows.

[0143] According to embodiments of this disclosure, the second processing module 620 includes a first integration unit, a generation unit, a normalization unit, a first mapping unit, and a second mapping unit.

[0144] The first integration unit is used to perform pixel-by-pixel integration on each first pixel row to obtain the total number of targets corresponding to the image and the integrated first pixel row.

[0145] The generation unit is used to generate an integrated target probability map based on the first pixel row after multiple integrations.

[0146] The normalization unit is used to normalize the integrated target probability distribution map so that the pixel value of each pixel in the target probability distribution map is within a first preset range.

[0147] The first mapping unit is used to perform a first mapping process on the normalized target probability distribution map to obtain a target grayscale image, wherein the pixel value of each pixel in the target grayscale image is within a second preset range.

[0148] The second mapping unit is used to perform a second mapping process on the target grayscale image using the color space to obtain a pseudo-color image, wherein the pseudo-color image represents the target heat map.

[0149] According to embodiments of this disclosure, the second processing module 620 further includes a filtering unit.

[0150] The filtering unit is used to perform threshold filtering on the target probability distribution map based on a preset pixel threshold to obtain a filtered target probability distribution map, and to perform pixel-by-pixel integration on each first pixel row of the filtered target probability distribution map.

[0151] According to embodiments of this disclosure, target statistics include the total number of targets.

[0152] According to embodiments of this disclosure, the target grayscale image includes a plurality of unconnected initial regions, wherein each initial region includes a plurality of clustered targets.

[0153] According to embodiments of this disclosure, the third processing module 630 includes an expansion unit and an corrosion unit.

[0154] The dilation unit is used to dilate any two initial regions, provided that the distance between the two initial regions meets a distance threshold, by using the structuring element in the morphological closing operation of the image to obtain a transition region, wherein the two dilated initial regions in the transition region partially overlap.

[0155] The erosion unit is used to erode the transition region using the structuring element in the image morphology closing operation to obtain a connected region. The difference between the image size of the connected region and the image size of the initial region satisfies a preset classification threshold, and the two eroded initial regions in the connected region partially overlap.

[0156] According to embodiments of this disclosure, the first processing module 610 includes an acquisition unit, a deletion unit, and an identification unit.

[0157] The acquisition unit is used to acquire an image acquired by an image acquisition device, wherein the resolution of the image is a preset resolution and the image includes at least one shadow area.

[0158] The deletion unit is used to delete the shadow area in the image if the pixel length or pixel width of the shadow area is greater than a preset classification threshold, thus obtaining the filtered image.

[0159] The recognition unit is used to identify the target type of the filtered image using a target recognition model, and obtain the target distribution probability map.

[0160] According to embodiments of this disclosure, the filtered image includes a target region that meets a preset classification threshold and other regions that do not meet the preset classification threshold. The target region includes at least one type of target of a first type. The target recognition model includes a target detection algorithm and a density estimation algorithm.

[0161] According to embodiments of this disclosure, the identification unit includes an identification subunit and a processing subunit.

[0162] The identification subunit is used to perform type identification processing on the target area using a target detection algorithm to obtain the type information and quantity information of the first type corresponding to the target area.

[0163] The processing subunit uses a density estimation algorithm to process each pixel in the target region and other regions to obtain the target probability corresponding to each pixel.

[0164] According to embodiments of this disclosure, the identification module 640 includes an obtaining unit and a second integrating unit.

[0165] The unit is used to process connected regions using a preset classification model to obtain the type information of targets within the connected regions. The preset classification model is constructed based on a convolutional neural network.

[0166] The second integration unit is used to perform pixel-by-pixel integration on each second pixel row in the connected region to obtain the number of targets corresponding to the connected region.

[0167] According to embodiments of this disclosure, the target recognition model is trained using a first training image and first label data, wherein the first training image includes a plurality of first training targets, and the first label data characterizes the position and type of the plurality of first training targets in the first training image.

[0168] The preset classification model is trained using the second training image and the second label data. The second training image includes the second training target, and the second label data represents the type of the second training target.

[0169] Any one or more of the modules, units, and subunits according to the embodiments of this disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, units, and subunits according to the embodiments of this disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, units, and subunits according to the embodiments of this disclosure can be at least partially implemented as hardware circuits, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented by hardware or firmware through any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in a suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, units, and subunits according to the embodiments of this disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0170] For example, any plurality of the first processing module 610, the second processing module 620, the third processing module 630, and the identification module 640 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first processing module 610, the second processing module 620, the third processing module 630, and the identification module 640 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first processing module 610, the second processing module 620, the third processing module 630, and the identification module 640 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0171] It should be noted that the image processing apparatus part in the embodiments of this disclosure corresponds to the image processing method part in the embodiments of this disclosure. The specific description of the image processing apparatus part is referred to in the image processing method part, and will not be repeated here.

[0172] Figure 7 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0173] like Figure 7As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from storage portion 708 into random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0174] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0175] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0176] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0177] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0178] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0179] According to embodiments of this disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0180] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the image processing methods provided in the embodiments of this disclosure.

[0181] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0182] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0183] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0185] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents without departing from the scope of this disclosure. Various substitutions and modifications can be made by those skilled in the art, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image processing method, characterized by, include: The image is processed using a target recognition model to obtain a target probability map and the type and quantity information of each type of target in the first type. The target probability map includes the target probability corresponding to each pixel. The first type is determined based on the pixel value of the target in the image and a preset classification threshold. The target probability map includes multiple first pixel rows. For each row of first pixels, pixel-wise integration is performed on the row to obtain the total number of targets corresponding to the image. After pixel-wise integration, the target probability map is normalized so that the pixel value of each pixel in the target probability map is within a first preset range. The normalized target probability map is then subjected to a first mapping process to obtain a target grayscale image, wherein the pixel value of each pixel in the target grayscale image is within a second preset range. The target grayscale image is then subjected to a second mapping process using a color space to obtain a pseudo-color image, wherein the pseudo-color image represents a target heatmap. A morphological closing operation is performed on the target grayscale image to obtain multiple connected regions, wherein each connected region represents a clustering region of a type of target in a second type, the second type being determined based on the pixel value of the target in the image and the preset classification threshold; For each connected region, a preset classification model is used to process the connected region to obtain the type information of the target within the connected region, wherein the preset classification model is constructed based on a convolutional neural network; For each second pixel row in the connected region, pixel-by-pixel integration is performed on the second pixel row to obtain the number of targets corresponding to the connected region.

2. The method of claim 1, wherein, Also includes: Based on the type and quantity information of each type of target in the first type and the type and quantity information of targets in multiple connected areas, the target statistics information of the image is generated. Based on the image, a display image is generated according to the target heatmap and the target statistics, wherein the display image includes labeling information of multiple types of targets of the first type; The displayed image is visualized.

3. The method according to claim 2, characterized in that, Before performing the pixel-by-pixel integration process, the following is also included: Based on a preset pixel threshold, the target distribution probability map is subjected to threshold filtering to obtain a filtered target distribution probability map, and then pixel-by-pixel integration is performed on the filtered target distribution probability map. The target statistics include the total number of the targets.

4. The method according to claim 1, characterized in that, The target grayscale image includes multiple unconnected initial regions, wherein each initial region includes multiple clustered targets; The step of performing an image morphological closing operation on the target grayscale image to obtain multiple connected regions includes: For any two initial regions, if the distance between the two initial regions meets a distance threshold, the two initial regions are dilated using the structuring element in the image morphological closing operation to obtain a transition region, wherein the two dilated initial regions in the transition region partially overlap. The transition region is eroded using the structuring element in the image morphological closing operation to obtain a connected region, wherein the difference between the image size of the connected region and the image size of the initial region satisfies a preset classification threshold, and the two eroded initial regions in the connected region partially overlap.

5. The method according to claim 1, characterized in that, The process of processing the image using a target recognition model to obtain a target probability distribution map includes: The image is acquired by an image acquisition device, wherein the resolution of the image is a preset resolution, and the image includes at least one shadow area; The target recognition model is used to identify the target type in the image to obtain the target probability distribution map; After obtaining the target distribution probability map, the process also includes: For each shadow region, if the pixel length or pixel width of the shadow region is greater than a preset classification threshold, the shadow region is deleted from the image to obtain a filtered image, and the target distribution probability map is generated based on the filtered image.

6. The method according to claim 5, characterized in that, The filtered image includes target regions that meet the preset classification threshold and other regions that do not meet the preset classification threshold. The target regions include at least one type of target of the first type. The target recognition model includes a target detection algorithm and a density estimation algorithm. The step of using the target recognition model to identify the target type of the filtered image to obtain the target distribution probability map includes: For the target region, the target detection algorithm is used to perform type identification processing on the targets in the target region to obtain the type information and quantity information of the first type corresponding to the target region; The density estimation algorithm is used to process each pixel in the target region and the other regions to obtain the target probability corresponding to each pixel.

7. The method according to claim 1, characterized in that, The target recognition model is trained using a first training image and first label data, wherein the first training image includes multiple first training targets, and the first label data represents the position and type of the multiple first training targets in the first training image; The preset classification model is trained using the second training image and the second label data, wherein the second training image includes the second training target and the second label data represents the type of the second training target.

8. An image processing apparatus, characterized in that, include: The first processing module is used to process the image using a target recognition model to obtain a target distribution probability map and the type and quantity information of each type of target in the first type. The target distribution probability map includes the target probability corresponding to each pixel. The first type is determined based on the pixel value of the target in the image and a preset classification threshold. The target distribution probability map includes multiple first pixel rows. The second processing module includes: The first integration unit is used to perform pixel-by-pixel integration on each first pixel row to obtain the total number of targets corresponding to the image and the integrated first pixel row. The generation unit is used to generate an integrated target probability map based on multiple integrated first pixel rows. The normalization unit is used to normalize the integrated target probability distribution map so that the pixel value of each pixel in the target probability distribution map is within a first preset range. The first mapping unit is used to perform a first mapping process on the normalized target distribution probability map to obtain a target grayscale image, wherein the pixel value of each pixel in the target grayscale image is within a second preset range. The second mapping unit is used to perform a second mapping process on the target grayscale image using the color space to obtain a pseudo-color image, wherein the pseudo-color image represents the target heat map; The third processing module is used to perform an image morphological closing operation on the target grayscale image to obtain multiple connected regions, wherein each connected region represents a clustering region of a type of target in the second type, and the second type is determined based on the pixel value of the target in the image and the preset classification threshold. The recognition module includes: The unit is used to process each connected region using a preset classification model to obtain the type information of the target within the connected region, wherein the preset classification model is constructed based on a convolutional neural network; The second integration unit is used to perform pixel-by-pixel integration on each second pixel row in the connected region to obtain the number of targets corresponding to the connected region.

Images