An image processing method, apparatus, electronic device, and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium, and the method comprises the steps of determining gradient feature information of a to-be-processed image in a preset gradient direction; the to-be-processed image comprises a target object; the gradient feature information in the preset gradient direction is related to the activity track of the target object in the preset gradient direction; performing multi-frequency domain filtering processing on the to-be-processed image according to the gradient feature information, to obtain a plurality of frequency domain one-to-one corresponding multiple frames of gray scale images; each frame of gray scale image in the multiple frames of gray scale images comprises the target object in the frequency domain feature information corresponding to each frame of gray scale image; performing super-pixel segmentation on the feature map fused from the multiple frames of gray scale images, to obtain the pixel region of the target object in the feature map. In this way, complete segmentation of the target object from the image can be realized, and the detection accuracy of the target object can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of computer technology, more and more scenarios require target detection, such as identifying moving objects in videos or identifying moving game characters in games.

[0003] In related technologies, computer devices use frame differencing to determine the position of moving targets. This involves subtracting the values ​​of two or more consecutive frames to obtain a difference image. Since the difference between background pixel values ​​is small or zero, while the difference between moving target pixel values ​​is large, the computer device can binarize the resulting difference image to detect the moving target.

[0004] However, the active areas of a moving target may overlap in two or more consecutive frames. The pixel values ​​of the overlapping parts are subtracted and the result is zero. The frame difference method cannot detect the overlapping parts, resulting in the moving target being fragmented into multiple regions or holes appearing inside the moving target in the final detection result. Therefore, the existing detection accuracy is not high. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides an image processing method, apparatus, electronic device, and storage medium:

[0006] According to a first aspect of this application, an image processing method is provided, comprising:

[0007] Determine the gradient feature information of the image to be processed in the preset gradient direction; the image to be processed includes the target object; the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction;

[0008] Based on the gradient feature information, the image to be processed is subjected to multi-frequency domain filtering to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains; each grayscale image in the multiple grayscale images includes the frequency domain feature information of the target object in each grayscale image;

[0009] Superpixel segmentation is performed on the feature map after fusing multiple grayscale images to obtain the pixel region of the target object in the feature map.

[0010] According to a second aspect of this application, an image processing apparatus is provided, comprising:

[0011] The determination module is used to determine the gradient feature information of the image to be processed in the preset gradient direction; the image to be processed includes the target object; the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction;

[0012] The filtering module is used to perform multi-frequency domain filtering on the image to be processed based on gradient feature information, so as to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains; each grayscale image in the multiple grayscale images includes the frequency domain feature information of the target object in each grayscale image;

[0013] The segmentation module is used to perform superpixel segmentation on the feature map after fusing multiple grayscale images to obtain the pixel region of the target object in the feature map.

[0014] According to a third aspect of this application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the image processing method as described in the first aspect.

[0015] According to a fourth aspect of this application, a computer storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the image processing method as described in the first aspect.

[0016] According to a fifth aspect of this application, a computer program product is provided, comprising at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is loaded and executed by a processor to implement the image processing method as described in the first aspect.

[0017] The image processing method, apparatus, electronic device, and storage medium provided in this application have the following technical effects:

[0018] When processing an image containing a target object, this application performs multi-frequency domain filtering on the image based on the gradient feature information of the image in a preset gradient direction to obtain multiple grayscale images. Since the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction, the multiple grayscale images can reflect the feature information of the target object in different frequency domains. Therefore, when segmenting the feature map obtained based on the multiple grayscale images, combined with the superpixel segmentation method, the target object can be completely segmented from the image, which can improve the detection accuracy of the target object. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0021] Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a preset gradient direction provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a process for determining the gradient feature information of each pixel according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a process for performing multi-frequency domain filtering on an image to be processed, provided in an embodiment of this application.

[0025] Figure 6 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of an image dimensionality reduction method provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of a superpixel segmentation process provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of a process for pre-segmenting feature maps after fusion of multiple grayscale images, provided in an embodiment of this application.

[0029] Figure 10 This is a schematic diagram illustrating a clustering effect provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of an image processing procedure provided in an embodiment of this application;

[0031] Figure 12 This is a block diagram of an image processing apparatus provided in an embodiment of this application;

[0032] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0035] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. This application environment may include a server 01 and a client 02. The server 01 and client 02 can be connected wirelessly or via a wired connection. In this application scenario, the server 01 can obtain an image to be processed from the client 02. The image to be processed includes a target object. The server 01 processes the image to detect the target object, so that either the server 01 or the client 02 can perform subsequent applications based on the detected target object.

[0036] The server 01 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory, etc. The server can provide backend services to the corresponding clients.

[0037] The client 02 can be a physical device such as a smartphone, computer (e.g., desktop computer, tablet, laptop), augmented reality (AR) / virtual reality (VR) device, digital assistant, smart voice interaction device (e.g., smart speaker), smart wearable device, smart home appliance, or in-vehicle terminal; or it can be software running on the physical device, such as a computer program. The operating system corresponding to the client can be Android, iOS (a mobile operating system developed by Apple), Linux, or Microsoft Windows.

[0038] It should be noted that, Figure 1 The example shown is only one application environment of the image processing method provided in this application. In practical applications, other application environments may also be included.

[0039] The following describes a specific embodiment of an image processing method according to this application. Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. This application provides the operational steps of the method described in the embodiment or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiment is merely one possible execution order among many and does not represent the only possible execution order. In actual systems or products, the method can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiment or the accompanying drawings.

[0040] Specific examples Figure 2 As shown, the method may include:

[0041] S201: Determine the gradient feature information of the image to be processed in the preset gradient direction; the image to be processed includes the target object; the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction.

[0042] In this embodiment, the image to be processed can be obtained by the server from a terminal, which can be an image acquisition device, such as a camera set on the terminal. Alternatively, the image to be processed can be generated by the server using a graphics renderer. Or, the image to be processed can be sent by another server requesting the current server to perform image processing.

[0043] In this embodiment, the image to be processed contains a target object, which is an object that matches the actual application scenario. For example, in a game application scenario, the image to be processed can be a scene in the game, and the target object can be a game character; or, in an autonomous driving application scenario, the image to be processed can be an image of the vehicle's environment taken by an onboard camera, and the target object can be a vehicle, pedestrian, obstacle, etc.

[0044] It should be noted that the main purpose of image processing in this application includes segmenting the target object from the image to be processed, and the actual content referred to by the target object is not limited. Therefore, as long as the means used are similar to the image processing method provided in the embodiments of this application, regardless of what kind of object the target object is or what acquisition channel the image to be processed is from, it is within the protection scope of this application.

[0045] In this embodiment, after the server obtains the image to be processed, it first determines the gradient feature information of the image to be processed in a preset gradient direction. The preset gradient direction can be determined based on the two-dimensional plane where the image to be processed is located, and correspondingly, the gradient feature information can be determined based on the pixel values ​​of all pixels in the image to be processed; here, the pixel values ​​can include red channel values, green channel values, and blue channel values, or the pixel values ​​can also include grayscale values.

[0046] In some possible embodiments, the preset gradient direction includes a first gradient direction and a second gradient direction; such as Figure 3 As shown, based on the two-dimensional plane where the image to be processed is located, a two-coordinate system xoy is established with the lower left corner of the image to be processed as the origin; then, the first gradient direction can point along the x-axis direction, and the second gradient direction can point along the y-axis direction.

[0047] Considering real-world application scenarios, such as in a shooting game, where characters frequently jump or crouch while dodging or attacking, the traditional frame difference method cannot adapt to situations where the target object undergoes rapid changes in movement, thus failing to effectively detect the target object.

[0048] Based on this, this application considers that the trajectory features of the target object's activities in the corresponding application scenario will be directly reflected in the pixel gradient features at the corresponding position on the image. Thus, this application uses the gradient feature information of the image to be processed in the preset gradient direction to associate it with the activity trajectory of the target object in the preset gradient direction, thereby first determining the gradient feature information of the image to be processed, so as to use the gradient feature information in subsequent steps to detect the target object from the image to be processed.

[0049] Accordingly, in one possible embodiment, the image processing method of this application may further include, for example, Figure 4The following steps are shown:

[0050] S401: Determine the grayscale information of each pixel in the image to be processed.

[0051] Here, the grayscale information of each pixel refers to the grayscale value of each pixel. The grayscale value of each pixel can be determined using relevant calculation methods based on the red, green, and blue channel values ​​of that pixel. In a specific example, the grayscale value of each pixel can be determined using the following formula (1):

[0052] Gray(i,j)=0.299*R(i,j)+0.587*G(i,j)+0.144*B(i,j)……(1)

[0053] Where Gray(i,j) represents the grayscale value of pixel (i,j); R(i,j) represents the red channel value; G(i,j) represents the green channel value; and B(i,j) represents the blue channel value.

[0054] S403: In the first gradient direction, determine the first gradient feature information of each pixel based on the grayscale information of at least one neighboring pixel.

[0055] Here, at least one pixel adjacent to each pixel refers to a pixel adjacent to each pixel in the first gradient direction; the first gradient feature information of each pixel refers to the first gradient feature value of each pixel, which can be calculated based on the gray value of at least one pixel adjacent to the pixel in the first gradient direction; usually, the first gradient feature value of the pixel can be obtained by subtracting the gray values ​​of the two pixels adjacent to the pixel in the first gradient direction.

[0056] Specifically, the aforementioned at least one pixel can include two pixels; such as Figure 3 As shown, the two pixels adjacent to pixel (i,j) in the first gradient direction include pixel (i+1,j) and pixel (i-1,j). Accordingly, the first gradient feature value of pixel (i,j) can be obtained by subtracting the gray value of pixel (i-1,j) from the gray value of pixel (i+1,j).

[0057] S405: In the second gradient direction, determine the second gradient feature information of each pixel based on the grayscale information of at least one neighboring pixel.

[0058] Similar to step S403 above, the second gradient feature information of each pixel refers to the second gradient feature value of each pixel, which can be calculated based on the gray value of at least one pixel adjacent to the pixel in the second gradient direction. Generally, the second gradient feature value of the pixel can be obtained by subtracting the gray values ​​of the two pixels adjacent to the pixel in the second gradient direction.

[0059] Specifically, the aforementioned at least one pixel can include two pixels; such as Figure 3 As shown, the two pixels adjacent to pixel (i,j) in the second gradient direction include pixel (i,j+1) and pixel (i,j-1). Accordingly, the first gradient feature value of pixel (i,j) can be obtained by subtracting the gray value of pixel (i,j-1) from the gray value of pixel (i,j+1).

[0060] S407: Based on the first gradient feature information and the second gradient feature information of each pixel, the gradient feature information of each pixel is obtained.

[0061] In this step, the first gradient feature information and the second gradient feature information of each pixel are used as the gradient feature information of each pixel; that is, the gradient feature information of each pixel includes the first gradient feature value calculated by step S403 and the second gradient feature value calculated by step S405.

[0062] In the above embodiments, by determining the gradient feature information of each pixel in the image to be processed, the gradient feature information of the image to be processed can be obtained. Thus, the server can use the gradient feature information of the image to be processed to perform subsequent processing.

[0063] S203: Based on the gradient feature information, perform multi-frequency domain filtering on the image to be processed to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains.

[0064] In this embodiment, the server uses the gradient feature information of the image to be processed to perform multi-frequency domain filtering processing on the image to be processed, and obtains multiple grayscale images with one-to-one correspondence in multiple frequency domains; wherein, each grayscale image in the multiple grayscale images reflects the image features of different frequency domains, and at the same time, each grayscale image includes the frequency domain feature information of the target object in the frequency domain corresponding to the grayscale image.

[0065] In this embodiment, multiple filters are designed to achieve multi-frequency domain filtering of the image to be processed.

[0066] In some possible embodiments, the filtering formula of the filter refers to the following formula (2):

[0067]

[0068] Among them, g k,s (x,y) represents the pixel after filtering by the filter parameter set [k,s]. Different filters correspond to different filter parameter sets, which will be described in detail below and will not be repeated here; (x′,y′) represents the pixel coordinates in the original image to be processed; f sRepresents pixel value; This indicates the phase; different filters use different phases.

[0069] The filtering formula for the above filter is based on Euler's formula: e ix =cos(x)+i*sin(x) is designed. In the field of image frequency domain filtering, corresponding filtering formulas can usually be designed based on Euler's formula according to different filtering purposes to achieve frequency domain filtering of two-dimensional images.

[0070] This application obtains multiple different filter parameter sets [k,s] by modifying the filter parameter set [k,s] in the above filter formula (2); based on the multiple different filter parameter sets [k,s], multiple filters are obtained, which are used to filter the image to be processed in multiple different frequency domains to obtain filtered images in different frequency domains; thus, multi-frequency domain filtering of the image to be processed is realized.

[0071] Furthermore, the aforementioned filter parameter set [k,s] is determined based on the gradient feature information of the image to be processed. Since the gradient feature information of the image to be processed is related to the movement trajectory of the target object, the grayscale image obtained after filtering the image to be processed based on the filter parameter set [k,s] mainly contains the frequency domain feature information of the target object in the corresponding frequency domain. Generally, frequency domain feature information can refer to the contour, texture, and other feature information of the target object.

[0072] Therefore, in some possible embodiments, the above-described multi-frequency domain filtering process on the image to be processed to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains may include, for example: Figure 5 The following steps are shown:

[0073] S501: Obtain multiple filter parameter groups; each filter parameter group includes a first filter parameter and a second filter parameter; the first filter parameter corresponds to the first gradient direction, and the second filter parameter corresponds to the second gradient direction.

[0074] Accordingly, in the above filter parameter set [k,s], k can represent the first filter parameter, and s can represent the second filter parameter. To ensure consistency in filtering across different frequency domains, the multiple first filter parameters in the multiple filter parameter sets are arranged in a first arithmetic progression, and the multiple second filter parameters are arranged in a second arithmetic progression.

[0075] In gaming scenarios, game characters typically move more frequently along the second gradient direction (y-axis). This corresponds to more pronounced or drastic changes in pixel values ​​along the y-axis compared to the x-axis in an image. Therefore, in a specific example, the difference between two adjacent first filter parameters can be smaller than the difference between two adjacent second filter parameters. This ensures that the filtering on the x-axis and y-axis is synchronized.

[0076] Here, this application combines actual game application scenarios and provides nine filtering parameter groups based on the gradient feature information of the game screen in actual game application scenarios: [0,0], [3,4], [6,8], [9,12], [12,16], [15,20], [18,24], [21,28] and [24,32].

[0077] S503: Based on each of the multiple filter parameter groups, filter the image to be processed to obtain the grayscale image corresponding to each filter parameter group.

[0078] In this step, as can be seen from the filtering formula (2) above, the filter is mainly based on the filtering parameter set [k,s] and phase. To filter the image to be processed, after determining multiple sets of filtering parameters, a corresponding number of phases can be determined. Here, this application also provides nine phases as an example, namely 0, π / 8, π / 4, π / 2, π, 5π / 4, 3π / 2, 7π / 4, and 2π. Thus, based on each set of filtering parameters [k,s] and the corresponding phase in the multiple sets of filtering parameters... By performing multi-frequency domain filtering on the image to be processed, multiple grayscale images with one-to-one correspondence in the frequency domain can be obtained.

[0079] In this embodiment of the application, after obtaining multiple grayscale images, the multiple grayscale images are fused together, and the server further processes the feature map after the multiple grayscale images are fused together.

[0080] In some possible embodiments, in order to reduce the computational load on the server, data dimensionality reduction is performed during the fusion process, that is, the feature map after the fusion of multiple grayscale images can be dimensionality-reduced.

[0081] Accordingly, before the next step, the image processing method of this application embodiment may further include, as follows: Figure 6 The following steps are shown:

[0082] S601: Fuse multiple grayscale images to obtain fused grayscale images.

[0083] In this context, fusion can refer to stitching. That is, directly stitching together multiple grayscale images to obtain a stitched multi-frame grayscale image.

[0084] S603: Perform dimensionality reduction processing on the fused multi-frame grayscale images to obtain a feature map; the feature map has the same dimension as the image to be processed.

[0085] In this step, the stitched multi-frame grayscale images are subjected to dimensionality reduction so that the dimension of the reduced feature map is the same as the dimension of the image to be processed. This allows the target object to be located in the feature map and then directly labeled at the corresponding position in the image to be processed.

[0086] In one specific embodiment, the grayscale image has the same dimension as the image to be processed, and the position of each pixel in the grayscale image is the same in the other grayscale image; therefore, during dimensionality reduction, for each pixel position in the feature map, a pixel at the corresponding position in one frame of grayscale images from multiple frames can be randomly selected as the pixel at that position in the feature map. For example, ... Figure 7 As shown, assuming that three grayscale images are obtained after multi-frequency domain filtering, during the dimensionality reduction process, the pixel at point K in the first grayscale image can be randomly selected from the three grayscale images and used as the pixel at point K' in the corresponding position in the feature map. Alternatively, the pixel at point M in the second grayscale image can be randomly selected from the three grayscale images and used as the pixel at point M' in the corresponding position in the feature map.

[0087] In another specific embodiment, a sparse dictionary can be constructed using the K-SVD algorithm, and then used to perform dimensionality reduction on the fused multi-frame grayscale images. Specifically, dictionary learning can also be simply called sparse coding, and it focuses on learning the dictionary D. From a matrix factorization perspective, the dictionary learning process is as follows: given a sample dataset Y, where each column of Y represents a sample; the goal of dictionary learning is to decompose the Y matrix into matrices D and X.

[0088] Y≈D*X

[0089] Meanwhile, to satisfy the constraints that the original matrix data X should be as sparse as possible, and that each column of D is a normalized vector, the dictionary D can be learned using the following objective function:

[0090] D,X=argmin D,X {||X||0}

[0091] st.||Y-DX|| 2 ≤ε

[0092] Where ε is the maximum value allowed for the reconstruction error.

[0093] Solving K-SVD is an iterative process. The objective function contains two unknown variables, D and X. One variable is fixed, and the other variable is updated. This process is repeated iteratively until convergence is achieved.

[0094] Finally, the learned dictionary D is used as a dimensionality reduction matrix to perform dimensionality reduction processing on the fused multi-frame grayscale images; in this way, the feature maps sparsed by the sparse dictionary can greatly preserve the feature information of the original image.

[0095] In the above embodiments, by significantly reducing the dimensionality of the fused multi-frame grayscale images to obtain the dimensionality-reduced feature map, most of the information after frequency domain filtering can be retained while reducing the complexity of subsequent image processing.

[0096] S205: Perform superpixel segmentation on the feature map after fusing multiple grayscale images to obtain the pixel region of the target object in the feature map.

[0097] In this embodiment of the application, after obtaining the feature map after fusing multiple grayscale images, the server performs superpixel segmentation on the feature map to obtain the pixel region of the target object in the feature map.

[0098] In some possible embodiments, during superpixel segmentation, the feature map is first pre-segmented. To address the issue of fragmented pixel regions corresponding to the target object after pre-segmentation, the pixels in the pre-segmented image are clustered to obtain the superpixels corresponding to the target object, i.e., the pixel regions of the target object in the feature map. Correspondingly, the aforementioned superpixel segmentation of the feature map after fusing multiple grayscale images to obtain the pixel regions of the target object in the feature map may include, for example... Figure 8 The following steps are shown:

[0099] S801: Pre-segment the feature map after fusing multiple grayscale images to obtain a pre-segmented image; the pre-segmented image includes multiple pixel regions.

[0100] In one specific embodiment, step S801 may include, for example: Figure 9 The following steps are shown:

[0101] S901: On the preset gradient direction, determine the gradient feature information of each pixel in the feature map based on the grayscale information of at least one pixel adjacent to each pixel in the feature map.

[0102] The preset gradient direction may include the first gradient direction and / or the second gradient direction described above. The process of determining the gradient feature information of each pixel in the feature map can refer to the process of determining the gradient feature information of each pixel in the image to be processed described above, and will not be repeated here.

[0103] S903: Based on the gradient feature information of each pixel, determine multiple edge pixels from the feature map.

[0104] In this context, the gradient feature information of each edge pixel among multiple edge pixels is greater than or equal to preset gradient feature information. The preset gradient feature information may include a first preset gradient feature value corresponding to a first gradient direction and / or a second preset gradient feature value corresponding to a second gradient direction.

[0105] Here, taking the preset gradient direction as including the first gradient direction and the second gradient direction as an example, the gradient feature information of each pixel in the feature map includes the first gradient feature value and the second gradient feature value, and the preset gradient feature information includes the first preset gradient feature value and the second preset gradient feature value.

[0106] Accordingly, the above-mentioned determination of multiple edge pixels from the feature map based on the gradient feature information of each pixel may specifically include: for each pixel in the feature map, if the first gradient feature value of the pixel is greater than or equal to the first preset gradient feature value, and the second gradient feature value of the pixel is greater than or equal to the second preset gradient feature value, then the pixel is determined as an edge pixel.

[0107] S905: Connects multiple edge pixels to obtain multiple pixel regions.

[0108] Edge pixels refer to pixels whose grayscale values ​​change drastically from those of their neighbors. Edge pixels typically indicate the edge contour of an object. Therefore, by determining the gradient characteristics of each pixel through the grayscale values ​​of its neighboring pixels, and then judging the magnitude of these gradient characteristics, the edge pixels of the object can be identified. Finally, by connecting the edge pixels, the edge contour of the object can be obtained. At the same time, the interior of the edge contour can form a closed pixel region, which may correspond to an object or a part of an object.

[0109] In addition, in other embodiments, pre-segmentation can also be achieved by using other pre-segmentation algorithms or machine models in the relevant field to pre-segment the feature map.

[0110] S803: Cluster multiple pixel regions to obtain the pixel regions of the target object in the feature map.

[0111] As mentioned earlier, each pixel region may correspond to a complete object. However, in practical applications, due to oversegmentation by the algorithm, in most cases each pixel region only corresponds to a part of an object, meaning a complete object is fragmented into multiple regions. Therefore, this application performs clustering on the pre-segmentation results, i.e., multiple pixel regions, to form superpixels, ultimately obtaining the pixel regions of the target object in the feature map.

[0112] In one specific embodiment, multiple pixel regions obtained from pre-segmentation are merged to form superpixels based on the fuzzy C-means clustering algorithm (FCM algorithm) to obtain the final segmentation result; the segmentation result includes the pixel region of the target object in the feature map.

[0113] The FCM algorithm combines fuzzy set knowledge with clustering algorithms, representing an improvement over hard clustering algorithms. It obtains the optimal solution and segmentation result by minimizing the objective function J. Its objective function is shown in formula (3):

[0114]

[0115] Where N is the total number of pixels in the image; c represents the number of clusters; m is a constant greater than 1 used to control the convergence speed. The larger the value of m, the faster the convergence. Generally speaking, m takes a value between 1.5 and 2.5; all pixels form a set X = {x1, x2, ..., x...} N}, x i u represents the grayscale value of the i-th pixel; ji Represents pixel x i The degree to which a class belongs to the j-th category, i.e., the membership degree, satisfies the following relationship:

[0116]

[0117] u ji ∈[0,1]

[0118] v j d(x) represents the gray value of the cluster center of the j-th category; i ,v j ) represents sample x i With cluster center v j Euclidean distance between ||x i -v j ||.

[0119] The cluster centers v are calculated iteratively. j and membership degree u ji The process continues until convergence, and finally, the category to which a pixel belongs is determined based on its membership degree.

[0120] Accordingly, based on the FCM algorithm, the above-mentioned clustering of multiple pixel regions to obtain the pixel regions of the target object in the feature map can include, for example: Figure 9 The following steps are shown:

[0121] S901: Determine the current cluster center and current superpixel region of the target object in the feature map.

[0122] Since the main purpose of this application is to segment the target object from the feature map, and based on the filtering process described above, the feature map mainly contains the features of the target object, this application can initialize only the cluster centers of the target object, i.e., determine the initial cluster centers of the target object in the feature map. Simultaneously, it can determine the initial superpixel region of the target object based on the initial cluster centers. This initial superpixel region can be the pixel region corresponding to the entire range of the feature map, or a partially defined pixel region in the feature map. In subsequent iterations, the current cluster centers and current superpixel regions of the target object in the feature map are continuously determined during the current iteration.

[0123] S903: Determine the membership degree of each pixel in the current superpixel region with the current cluster center, and obtain the current membership degree set.

[0124] In each iteration, the membership degree of each pixel in the current superpixel region with the current cluster center is determined. This membership degree represents the degree to which the pixel belongs to the target object.

[0125] S905: Determine the current clustering measurement data based on the current membership set, the current cluster center, and the current superpixel region.

[0126] In each iteration, the current clustering metric is determined based on the current membership set, the current cluster center, and the current superpixel region. Here, the current clustering metric is the function value of the objective function shown in formula (3) above.

[0127] S907: Iterate over the current cluster center and the current superpixel region to obtain the current membership set, the current cluster center, and the current superpixel region after iteration. Repeat the steps: Based on the current membership set, the current cluster center, and the current superpixel region, determine the current cluster measurement data.

[0128] S909: Until the iteration stop condition is met, the current superpixel region corresponding to the current cluster measurement data that meets the preset measurement data during the iteration process is determined as the pixel region of the target object in the feature map.

[0129] The iteration stopping conditions can include stopping iteration when the number of iterations reaches a preset number, or stopping iteration when the current cluster measurement data reaches a preset number of data.

[0130] In a specific example, the iteration stopping condition is to stop iterating when the preset number of iterations is reached. For example, the number of iterations can be 100. When iterating to the 100th iteration, the current cluster measure with the smallest value is determined from the 100 current cluster measure data corresponding to the 100 iterations. This current cluster measure with the smallest value may be the current cluster measure data corresponding to the 100th iteration or the current cluster measure data corresponding to the 98th iteration. This current cluster measure data with the smallest value is determined as the current cluster measure data that meets the preset measure data, and the current superpixel region determined in this iteration is determined as the pixel region of the target object in the feature map.

[0131] In another specific example, the iteration stops when the current clustering metric reaches a preset threshold. Let's continue with the example of 100 iterations. Upon reaching 100 iterations, the current clustering metric corresponding to the 100th iteration is directly determined as the current clustering metric that satisfies the preset threshold, and the current superpixel region determined during the 100th iteration is determined as the pixel region of the target object in the feature map.

[0132] like Figure 10 As shown, Figure 10 This is a schematic diagram illustrating a clustering effect provided in an embodiment of this application. Figure 10 The leftmost image in the table shows the original image to be processed. Figure 10 The intermediate image shown is the pre-segmented image. Figure 10 The rightmost image in the diagram shows the image after clustering. As can be seen from the diagram, clustering can further cluster multiple pixel regions into superpixels, thereby achieving the effect of completely separating the target object from the background.

[0133] The following collection Figure 11 This application provides a comprehensive description of the image processing steps provided in its embodiments. For example... Figure 11 As shown, firstly, the server processes the acquired image to be processed through multiple filters (QF1, QF2, ..., QF...). n The process involves filtering the images in different frequency domains to obtain multiple filtered images. These images reflect the frequency domain features of the target object in different frequency domains. Then, the server performs dimensionality reduction on the feature map after fusing the multiple filtered images to obtain a dimensionality-reduced feature map, which reduces the amount of image data processing required for subsequent pre-segmentation and clustering. Next, the dimensionality-reduced feature map is pre-segmented to obtain a pre-segmented image, achieving the initial extraction of the target object. Finally, the pre-segmented image is clustered to complete the final segmentation of the target object.

[0134] In summary, the image processing method provided in this application, when processing an image containing a target object, performs multi-frequency domain filtering on the image to be processed based on the gradient feature information of the image in a preset gradient direction, obtaining multiple frames of grayscale images. Since the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction, the multiple frames of grayscale images can reflect the feature information of the target object in different frequency domains. Therefore, when segmenting the feature map obtained based on the multiple frames of grayscale images, combined with the superpixel segmentation method, the target object can be completely segmented from the image, improving the detection accuracy of the target object. The image processing method of this application embodiment has a good effect on the recognition and trajectory determination of game characters when applied to game scenes.

[0135] This application also provides an image processing apparatus, such as... Figure 12 As shown, the image processing apparatus 120 includes:

[0136] The determining module 1201 is used to determine the gradient feature information of the image to be processed in a preset gradient direction; the image to be processed includes a target object; the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction;

[0137] The filtering module 1202 is used to perform multi-frequency domain filtering on the image to be processed based on gradient feature information to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains; each grayscale image in the multiple grayscale images includes the frequency domain feature information of the target object in each grayscale image.

[0138] The segmentation module 1203 is used to perform superpixel segmentation on the feature map after the fusion of multiple grayscale images to obtain the pixel region of the target object in the feature map.

[0139] In some possible embodiments, the preset gradient direction includes a first gradient direction and a second gradient direction; the determining module 1201 is further configured to determine the grayscale information of each pixel in the image to be processed; in the first gradient direction, determine the first gradient feature information of each pixel based on the grayscale information of at least one neighboring pixel of each pixel; in the second gradient direction, determine the second gradient feature information of each pixel based on the grayscale information of at least one neighboring pixel of each pixel; and obtain the gradient feature information of each pixel based on the first gradient feature information and the second gradient feature information of each pixel.

[0140] In some possible embodiments, the filtering module 1202 is further configured to acquire multiple filtering parameter groups; each filtering parameter group includes a first filtering parameter and a second filtering parameter; the first filtering parameter corresponds to the first gradient direction, and the second filtering parameter corresponds to the second gradient direction; the multiple first filtering parameters are in a first arithmetic progression relationship, and the multiple second filtering parameters are in a second arithmetic progression relationship; the difference between two adjacent first filtering parameters is less than the difference between two adjacent second filtering parameters; based on each filtering parameter group, the image to be processed is filtered to obtain a grayscale image corresponding to each filtering parameter group.

[0141] In some possible embodiments, the apparatus further includes:

[0142] The dimensionality reduction module is used to fuse the multi-frame grayscale images to obtain fused multi-frame grayscale images; and to perform dimensionality reduction processing on the fused multi-frame grayscale images to obtain the feature map; the feature map has the same dimension as the image to be processed.

[0143] In some possible embodiments, the segmentation module 1203 includes:

[0144] The pre-segmentation submodule is used to pre-segment the feature map after the fusion of the multi-frame grayscale images to obtain a pre-segmented image; the pre-segmented image includes multiple pixel regions.

[0145] The clustering submodule is used to cluster the multiple pixel regions to obtain the superpixel region of the target object in the feature map.

[0146] In some possible embodiments, the pre-segmentation submodule is further configured to determine gradient feature information of each pixel in the feature map based on the grayscale information of at least one pixel adjacent to each pixel in the feature map along the preset gradient direction; determine multiple edge pixels from the feature map based on the gradient feature information of each pixel; the gradient feature information of each edge pixel in the multiple edge pixels is greater than or equal to the preset gradient feature information; and connect the multiple edge pixels to obtain the multiple pixel regions.

[0147] In some possible embodiments, the clustering submodule is further configured to: determine the current cluster center and the current superpixel region of the target object in the feature map; determine the membership degree of each pixel in the current superpixel region with the current cluster center to obtain a current membership degree set; determine the current clustering metric data based on the current membership degree set, the current cluster center, and the current superpixel region; iterate over the current cluster center and the current superpixel region to obtain the iterated current membership degree set, the current cluster center, and the current superpixel region; repeat the step of determining the current clustering metric data based on the current membership degree set, the current cluster center, and the current superpixel region; until the iteration stop condition is met, determine the current superpixel region corresponding to the current clustering metric data that meets the preset metric data during the iteration process as the superpixel region of the target object in the feature map.

[0148] It should be noted that the apparatus and method embodiments described in the device embodiments are based on the same inventive concept.

[0149] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the image processing method provided in the above method embodiments.

[0150] Furthermore, Figure 13 A schematic diagram of the hardware structure of an electronic device for implementing the image processing method provided in the embodiments of this application is shown. The electronic device may participate in or include the classification device for the operation paths provided in the embodiments of this application. Figure 13 As shown, the electronic device 100 may include one or more processors 1002 (shown as 1002a, 1002b, ..., 1002n in the figure) (processor 1002 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 13 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 100 may also include... Figure 13 The more or fewer components shown, or having the same Figure 13 The different configurations shown.

[0151] It should be noted that the aforementioned one or more processors 1002 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the electronic device 100 (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0152] The memory 1004 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method described in the embodiments of this application. The processor 1002 executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, thereby implementing the aforementioned image processing method. The memory 1004 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include memory remotely located relative to the processor 1002, and these remote memories can be connected to the electronic device 100 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0153] The transmission device 1006 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 100. In one example, the transmission device 1006 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In one embodiment, the transmission device 1006 may be a radio frequency (RF) module for wireless communication with the Internet.

[0154] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows a user to interact with the user interface of the electronic device 100 (or mobile device).

[0155] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing an image processing method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the image processing method provided in the above method embodiments.

[0156] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0157] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0158] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and electronic device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0159] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0160] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: The gradient feature information of the image to be processed is determined in a preset gradient direction; the image to be processed includes a target object; the gradient feature information in the preset gradient direction is related to the movement trajectory of the target object in the preset gradient direction; the preset gradient direction includes a first gradient direction and a second gradient direction. Based on the gradient feature information, the image to be processed is subjected to multi-frequency domain filtering to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains. Each grayscale image in the multi-frame grayscale image includes the frequency domain feature information of the target object in each corresponding grayscale image; Superpixel segmentation is performed on the feature map after fusing the multi-frame grayscale images to obtain the superpixel region of the target object in the feature map; The step of performing multi-frequency domain filtering on the image to be processed to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains includes: Multiple filter parameter groups are obtained; each filter parameter group includes a first filter parameter and a second filter parameter; the first filter parameter corresponds to the first gradient direction, and the second filter parameter corresponds to the second gradient direction; the multiple first filter parameters are in a first arithmetic progression relationship, and the multiple second filter parameters are in a second arithmetic progression relationship; the difference between two adjacent first filter parameters is less than the difference between two adjacent second filter parameters. Based on each of the plurality of filter parameter groups, the image to be processed is filtered to obtain a grayscale image corresponding to each filter parameter group.

2. The image processing method according to claim 1, characterized in that, Before performing multi-frequency domain filtering on the image to be processed based on the gradient feature information to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains, the method further includes: Determine the grayscale information of each pixel in the image to be processed; In the first gradient direction, the first gradient feature information of each pixel is determined based on the grayscale information of at least one neighboring pixel of each pixel; In the second gradient direction, the second gradient feature information of each pixel is determined based on the grayscale information of at least one neighboring pixel of each pixel; The gradient feature information of each pixel is obtained based on the first gradient feature information and the second gradient feature information of each pixel.

3. The image processing method according to claim 1, characterized in that, The method further includes performing superpixel segmentation on the feature map after fusing the multi-frame grayscale images to obtain the target object before the superpixel region in the feature map. The multiple grayscale images are fused to obtain a fused multiple grayscale image; The fused multi-frame grayscale images are subjected to dimensionality reduction processing to obtain the feature map; the feature map has the same dimension as the image to be processed.

4. The image processing method according to claim 1 or 3, characterized in that, The step of performing superpixel segmentation on the feature map after fusing the multi-frame grayscale images to obtain the superpixel region of the target object in the feature map includes: The feature map after fusing the multi-frame grayscale images is pre-segmented to obtain a pre-segmented image; the pre-segmented image includes multiple pixel regions. Clustering is performed on the multiple pixel regions to obtain the superpixel regions of the target object in the feature map.

5. The image processing method according to claim 4, characterized in that, The step of pre-segmenting the feature map after fusing the multi-frame grayscale images to obtain a pre-segmented image includes: In the preset gradient direction, the gradient feature information of each pixel in the feature map is determined based on the grayscale information of at least one pixel adjacent to each pixel in the feature map; Based on the gradient feature information of each pixel, multiple edge pixels are determined from the feature map; the gradient feature information of each edge pixel among the multiple edge pixels is greater than or equal to a preset gradient feature information; The multiple edge pixels are connected to obtain the multiple pixel regions.

6. The image processing method according to claim 4, characterized in that, The step of clustering the plurality of pixel regions to obtain the superpixel region of the target object in the feature map includes: Determine the current cluster center and current superpixel region of the target object in the feature map; Determine the membership degree between each pixel in the current superpixel region and the current cluster center to obtain the current membership degree set; Based on the current membership set, the current cluster center, and the current superpixel region, determine the current clustering measurement data; Iterate over the current cluster center and the current superpixel region to obtain the iterated current membership set, current cluster center and current superpixel region. Repeat the following steps: determine the current clustering measurement data based on the current membership set, current cluster center and current superpixel region. Until the iteration stop condition is met, the current superpixel region corresponding to the current cluster measurement data that meets the preset measurement data during the iteration process is determined as the superpixel region of the target object in the feature map.

7. An image processing apparatus, characterized in that, include: A determining module is used to determine gradient feature information of an image to be processed in a preset gradient direction; the image to be processed includes a target object; the gradient feature information in the preset gradient direction is related to the activity trajectory of the target object in the preset gradient direction; the preset gradient direction includes a first gradient direction and a second gradient direction; The filtering module is used to perform multi-frequency domain filtering on the image to be processed based on the gradient feature information to obtain multiple grayscale images with one-to-one correspondence in multiple frequency domains. Each grayscale image in the multi-frame grayscale image includes the frequency domain feature information of the target object in each corresponding grayscale image; The segmentation module is used to perform superpixel segmentation on the feature map after the fusion of the multi-frame grayscale images to obtain the superpixel region of the target object in the feature map. The filtering module is further configured to acquire multiple filtering parameter groups; each filtering parameter group includes a first filtering parameter and a second filtering parameter; the first filtering parameter corresponds to the first gradient direction, and the second filtering parameter corresponds to the second gradient direction; the multiple first filtering parameters are in a first arithmetic progression relationship, and the multiple second filtering parameters are in a second arithmetic progression relationship; the difference between two adjacent first filtering parameters is less than the difference between two adjacent second filtering parameters. Based on each of the plurality of filter parameter groups, the image to be processed is filtered to obtain a grayscale image corresponding to each filter parameter group.

8. The apparatus according to claim 7, characterized in that, The determining module is further configured to determine the grayscale information of each pixel in the image to be processed; in the first gradient direction, determine the first gradient feature information of each pixel based on the grayscale information of at least one neighboring pixel of each pixel; in the second gradient direction, determine the second gradient feature information of each pixel based on the grayscale information of at least one neighboring pixel of each pixel. The gradient feature information of each pixel is obtained based on the first gradient feature information and the second gradient feature information of each pixel.

9. The apparatus according to claim 7, characterized in that, The device further includes: The dimensionality reduction module is used to fuse the multi-frame grayscale images to obtain fused multi-frame grayscale images; and to perform dimensionality reduction processing on the fused multi-frame grayscale images to obtain the feature map; the feature map has the same dimension as the image to be processed.

10. The apparatus according to claim 7 or 9, characterized in that, The segmentation module includes: The pre-segmentation submodule is used to pre-segment the feature map after the fusion of the multi-frame grayscale images to obtain a pre-segmented image; the pre-segmented image includes multiple pixel regions. The clustering submodule is used to cluster the multiple pixel regions to obtain the superpixel region of the target object in the feature map.

11. The apparatus according to claim 10, characterized in that, The pre-segmentation submodule is further configured to determine the gradient feature information of each pixel in the feature map based on the grayscale information of at least one pixel adjacent to each pixel in the feature map along the preset gradient direction. Based on the gradient feature information of each pixel, multiple edge pixels are determined from the feature map; the gradient feature information of each edge pixel is greater than or equal to a preset gradient feature information; the multiple edge pixels are connected to obtain the multiple pixel regions.

12. The apparatus according to claim 10, characterized in that, The clustering submodule is also used to determine the current cluster center and the current superpixel region of the target object in the feature map; Determine the membership degree between each pixel in the current superpixel region and the current cluster center to obtain the current membership degree set; Based on the current membership set, the current cluster center, and the current superpixel region, determine the current clustering measurement data; Iterate over the current cluster center and the current superpixel region to obtain the iterated current membership set, current cluster center and current superpixel region. Repeat the following steps: determine the current clustering measurement data based on the current membership set, current cluster center and current superpixel region. Until the iteration stop condition is met, the current superpixel region corresponding to the current cluster measurement data that meets the preset measurement data during the iteration process is determined as the superpixel region of the target object in the feature map.

13. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the image processing method as described in any one of claims 1-6.

14. A computer storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1-6.

15. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1-6.

Images