Image processing method, device and equipment

Abstract

The present application provides an image processing method, apparatus, and device, which includes: dividing multiple frames of initial images in an image sequence corresponding to a target object into a first type of initial images to be accelerated and a second type of initial images not to be accelerated based on a target frame interval; for the first type of initial images, determining a target area corresponding to the target object in the first type of initial images based on a foreground mask image corresponding to a candidate initial image; wherein the candidate initial image is an initial image in the image sequence that is located before the first type of initial image; intercepting a target sub-image corresponding to the target area from the first type of initial image; and obtaining a foreground image corresponding to the target object based on the target sub-image. Through the technical solution of the present application, the amount of calculation can be significantly reduced, the image cutout speed can be accelerated, and real-time requirements can be met.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus and device. Background Art

[0002] For a frame of initial image, the area of ​​interest in the initial image is called the foreground area, and the other areas outside the foreground area are called the background area. In order to distinguish the foreground area from the background area, the initial image needs to be cut out to obtain the foreground area in the initial image. In order to cut out the initial image, the initial image can be input into a deep learning network model, and the deep learning network model cuts out the initial image to obtain the foreground area in the initial image. For example, the deep learning network model determines the probability that each pixel in the initial image belongs to the foreground area. Based on the probability that each pixel belongs to the foreground area, the area composed of pixels with a relatively high probability is used as the foreground area in the initial image.

[0003] Since the video file includes a large number of initial images, when the video file is cut out, each frame of the initial image in the video file is cut out, and the cutout of each frame of the initial image requires a high amount of calculation. Therefore, when a large number of initial images in the video file are cut out, the amount of calculation is very large, the cutout speed is relatively slow, and it takes a long time to complete the cutout, which cannot meet the real-time requirements. Summary of the Invention

[0004] The present application provides an image processing method, the method comprising:

[0005] Based on the target frame interval, multiple frames of initial images in the image sequence corresponding to the target object are divided into a first type of initial images to be accelerated and a second type of initial images not to be accelerated;

[0006] For the first type of initial image, determining a target area corresponding to the target object in the first type of initial image based on a foreground mask image corresponding to a candidate initial image; wherein the candidate initial image is an initial image located before the first type of initial image in the image sequence;

[0007] intercepting a target sub-image corresponding to the target area from the first type of initial images;

[0008] A foreground image corresponding to the target object is acquired based on the target sub-image.

[0009] The present application provides an image processing device, comprising:

[0010] a division module, configured to divide, based on a target frame interval, a plurality of initial image frames in an image sequence corresponding to a target object into a first category of initial images to be accelerated and a second category of initial images not to be accelerated;

[0011] a determination module configured to determine, for the first type of initial image, a target region corresponding to the target object in the first type of initial image based on a foreground mask image corresponding to a candidate initial image; the candidate initial image being an initial image preceding the first type of initial image in the image sequence;

[0012] An acquisition module is configured to intercept a target sub-image corresponding to the target area from the first type of initial image; and acquire a foreground image corresponding to the target object based on the target sub-image.

[0013] The present application provides an image processing device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of the present application.

[0014] It can be seen from the above technical solution that in the embodiment of the present application, multiple frames of initial images in an image sequence (such as a video file) are divided into a first type of initial images to be accelerated and a second type of initial images that are not accelerated. For the first type of initial images, the target area corresponding to the target object in the first type of initial images is determined, and the target sub-image corresponding to the target area is cut out to obtain a foreground image. Since the target sub-image is a sub-image in the first type of initial image, that is, the size of the target sub-image is smaller than the size of the first type of initial image, the amount of calculation for cutting out the target sub-image is smaller than the amount of calculation for cutting out the first type of initial image, thereby reducing the amount of calculation. When the above method is used to cut out multiple frames of the first type of initial images, the amount of calculation can be significantly reduced, the cutting out speed is relatively fast, the cutting out speed is accelerated, and the cutting out can be completed in a relatively short time, thereby meeting the real-time requirements. It can make full use of the continuity information between images, determine the target area corresponding to the target object in the first type of initial image, and adaptively change the cutout area, thereby automatically reducing the amount of calculation and speeding up the calculation speed, and well guaranteeing the high-precision requirements of cutouts. It does not need to increase hardware costs, can adaptively accelerate cutouts, and perform accelerated processing, which can ensure global real-time performance. For devices with poor performance, a relatively smooth and high-precision cutout effect can also be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0016] Figure 1 is a flowchart of an image processing method in one embodiment of the present application;

[0017] Figure 2 is a flowchart of an image processing method in one embodiment of the present application;

[0018] Figure 3A is a schematic diagram of the second type of initial image;

[0019] Figure 3B is a schematic diagram of the foreground image;

[0020] Figure 3C is a schematic diagram of the foreground mask image;

[0021] Figure 4 It is a schematic diagram of the total pixel values ​​of each column of the foreground mask image;

[0022] Figure 5A This is a schematic diagram of searching from the outside in;

[0023] Figure 5B It is a schematic diagram of using jump search;

[0024] Figures 6A-6E This is a schematic diagram of the processing process of the first type of initial image;

[0025] Figure 7 is a schematic structural diagram of an image processing device in one embodiment of the present application;

[0026] Figure 8 This is a hardware structure diagram of an image processing device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" used may also be interpreted as "at the time of" or "when" or "in response to determining".

[0029] The present application provides an image processing method, which can be applied to any type of device without any restriction on the type of device, such as a device with poor performance or a device with good performance. Figure 1 FIG. 1 is a flow chart of the image processing method, which may include the following steps:

[0030] Step 101: Divide multiple frames of initial images in an image sequence corresponding to a target object into a first category of initial images to be accelerated and a second category of initial images not to be accelerated based on a target frame interval.

[0031] Step 102: for the first type of initial image, determine the target area corresponding to the target object in the first type of initial image based on the foreground mask image corresponding to the candidate initial image; wherein the candidate initial image may be the initial image located before the first type of initial image in the image sequence.

[0032] Exemplarily, the pixel value corresponding to the background area of ​​the candidate initial image in the foreground mask image is a first value, and the pixel value corresponding to the foreground area of ​​the candidate initial image in the foreground mask image is greater than the first value; on this basis, the target area corresponding to the target object in the first type of initial image is determined based on the foreground mask image corresponding to the candidate initial image, which may include but is not limited to: based on the total value of pixels in each row of the foreground mask image, the upper and lower boundaries are determined, the total value of pixels in the upper boundary is greater than a preset threshold, and the total value of pixels in the lower boundary is greater than the preset threshold; based on the total value of pixels in each column of the foreground mask image, the left and right boundaries are determined, the total value of pixels in the left boundary is greater than the preset threshold, and the total value of pixels in the right boundary is greater than the preset threshold; the target area is determined based on the upper boundary, lower boundary, left boundary and right boundary.

[0033] Among them, based on the total pixel values ​​of each row of the foreground mask image, the upper boundary and the lower boundary are determined, which may include but is not limited to: traversing a first target row from the first row on the upper side of the foreground mask image, the first target row is the first row on the upper side whose total pixel value is greater than a preset threshold, and the first target row is determined as the upper boundary; traversing a second target row from the first row on the lower side of the foreground mask image, the second target row is the first row on the lower side whose total pixel value is greater than a preset threshold, and the second target row is determined as the lower boundary.

[0034] Among them, based on the total pixel values ​​of each column of the foreground mask image, the left boundary and the right boundary are determined, which may include but is not limited to: traversing a first target column from the first column on the left side of the foreground mask image, the first target column is the first column on the left side whose total pixel value is greater than a preset threshold, and the first target column is determined as the left boundary; traversing a second target column from the first column on the right side of the foreground mask image, the second target column is the first column on the right side whose total pixel value is greater than a preset threshold, and the second target column is determined as the right boundary.

[0035] In one possible embodiment, traversing the first target row from the first row on the upper side of the foreground mask image may include, but is not limited to, determining whether the total pixel value of the first row on the upper side is greater than a preset threshold; if so, determining the first row on the upper side as the first target row; if not, skipping K rows based on the first row on the upper side to obtain a first initial row, and determining whether the total pixel value of the first initial row is greater than a preset threshold, where K is a positive integer greater than 1. If not, skipping K rows based on the first initial row, updating the row after skipping K rows as the first initial row, and returning to the operation of determining whether the total pixel value of the first initial row is greater than the preset threshold. If so, backtracking K-1 rows based on the first initial row to obtain a second initial row, and determining whether the total pixel value of the second initial row is greater than the preset threshold; if so, determining the second initial row as the first target row; if not, updating the row next to the second initial row as the second initial row, and returning to the operation of determining whether the total pixel value of the second initial row is greater than the preset threshold.

[0036] In one possible embodiment, traversing the second target row from the first row at the bottom of the foreground mask image may include, but is not limited to, determining whether the total pixel value of the first row at the bottom is greater than a preset threshold; if so, determining the first row at the bottom as the second target row; if not, skipping K rows based on the first row at the bottom to obtain a third initial row, and determining whether the total pixel value of the third initial row is greater than a preset threshold, where K is a positive integer greater than 1. If not, skipping K rows based on the third initial row, updating the row after skipping K rows as the third initial row, and returning to the operation of determining whether the total pixel value of the third initial row is greater than the preset threshold. If so, backtracking K-1 rows based on the third initial row to obtain a fourth initial row, and determining whether the total pixel value of the fourth initial row is greater than the preset threshold; if so, determining the fourth initial row as the second target row; if not, updating the row above the fourth initial row as the fourth initial row, and returning to the operation of determining whether the total pixel value of the fourth initial row is greater than the preset threshold.

[0037] In one possible embodiment, traversing the first target column from the first column on the left side of the foreground mask image may include, but is not limited to, determining whether the total pixel value of the first column on the left side is greater than a preset threshold; if so, determining the first column on the left side as the first target column; if not, skipping K columns based on the first column on the left side to obtain a first initial column, and determining whether the total pixel value of the first initial column is greater than a preset threshold, where K is a positive integer greater than 1. If not, skipping K columns based on the first initial column, updating the column after skipping K columns to the first initial column, and returning to the operation of determining whether the total pixel value of the first initial column is greater than the preset threshold. If so, backtracking K-1 columns based on the first initial column to obtain a second initial column, and determining whether the total pixel value of the second initial column is greater than the preset threshold; if so, determining the second initial column as the first target column; if not, updating the column to the right of the second initial column to the second initial column, and returning to the operation of determining whether the total pixel value of the second initial column is greater than the preset threshold.

[0038] In one possible embodiment, traversing the second target column from the first column on the right side of the foreground mask image may include, but is not limited to, determining whether the total pixel value of the first column on the right side is greater than a preset threshold; if so, determining the first column on the right side as the second target column; if not, skipping K columns based on the first column on the right side to obtain a third initial column, and determining whether the total pixel value of the third initial column is greater than a preset threshold, where K is a positive integer greater than 1. If not, skipping K columns based on the third initial column, updating the column after skipping K columns to the third initial column, and returning to the operation of determining whether the total pixel value of the third initial column is greater than the preset threshold. If so, backtracking K-1 columns based on the third initial column to obtain a fourth initial column, and determining whether the total pixel value of the fourth initial column is greater than the preset threshold; if so, determining the fourth initial column as the second target column; if not, updating the column to the left of the fourth initial column to the fourth initial column, and returning to the operation of determining whether the total pixel value of the fourth initial column is greater than the preset threshold.

[0039] Step 103: intercepting a target sub-image corresponding to the target area from the first type of initial image.

[0040] Step 104: Acquire a foreground image corresponding to the target object based on the target sub-image.

[0041] Exemplarily, the target sub-image can be input into the target network model to obtain a foreground area in the target sub-image, and a foreground image corresponding to the target object is generated based on the foreground area.

[0042] For example, after dividing the multiple frames of initial images into a first category of initial images to be accelerated and a second category of initial images that are not accelerated, for the second category of initial images, a foreground image corresponding to the target object can be directly obtained based on the second category of initial images (i.e., the entire second category of initial images, not a sub-image). For example, the second category of initial images can be input into the target network model to obtain a foreground region in the second category of initial images, and a foreground image corresponding to the target object is generated based on the foreground region.

[0043] In a possible implementation, a method for obtaining the target frame interval may include, but is not limited to: determining the configured initial frame interval as the target frame interval; or, adjusting the initial frame interval based on the motion state corresponding to the target object, and determining the adjusted frame interval as the target frame interval.

[0044] Exemplarily, adjusting the initial frame interval based on the motion state corresponding to the target object may include: determining the motion state corresponding to the target object based on the difference between target boundaries of M foreground mask images corresponding to the M initial images; M is a positive integer greater than 1, the M initial images are initial images preceding the first type of initial images in the image sequence, and the target boundaries include at least one of an upper boundary, a lower boundary, a left boundary, and a right boundary. If the motion state indicates that the target object is in a stable state, increasing the initial frame interval; if the motion state indicates that the target object is in an unstable state, decreasing the initial frame interval.

[0045] It can be seen from the above technical solution that in the embodiment of the present application, multiple frames of initial images in an image sequence (such as a video file) are divided into a first type of initial images to be accelerated and a second type of initial images that are not accelerated. For the first type of initial images, the target area corresponding to the target object in the first type of initial images is determined, and the target sub-image corresponding to the target area is cut out to obtain a foreground image. Since the target sub-image is a sub-image in the first type of initial image, that is, the size of the target sub-image is smaller than the size of the first type of initial image, the amount of calculation for cutting out the target sub-image is smaller than the amount of calculation for cutting out the first type of initial image, thereby reducing the amount of calculation. When the above method is used to cut out multiple frames of the first type of initial images, the amount of calculation can be significantly reduced, the cutting out speed is relatively fast, the cutting out speed is accelerated, and the cutting out can be completed in a relatively short time, thereby meeting the real-time requirements. It can make full use of the continuity information between images, determine the target area corresponding to the target object in the first type of initial image, and adaptively change the cutout area, thereby automatically reducing the amount of calculation and speeding up the calculation speed, and well guaranteeing the high-precision requirements of cutouts. It does not need to increase hardware costs, can adaptively accelerate cutouts, and perform accelerated processing, which can ensure global real-time performance. For devices with poor performance, a relatively smooth and high-precision cutout effect can also be achieved.

[0046] The above technical solutions of the embodiments of the present application are described below in conjunction with specific application scenarios.

[0047] When cutting out a video file, it is necessary to cut out each frame of the initial image in the video file, that is, to cut out the entire single frame of the initial image. Since cutting out each frame of the initial image requires a high amount of calculation, when cutting out the video file frame by frame, the amount of calculation is very large, the cutting out speed is relatively slow, and it takes a long time to complete the cutting out. On some lower-end devices, it may not meet the real-time requirements.

[0048] In response to the above findings, in this embodiment, the continuity information between the initial images in the video file is fully utilized to determine the target area corresponding to the target object in the initial image. The cutout area (i.e., the target area) is adaptively changed in combination with the motion changes of the target object. Cutout is performed based on the target area in the initial image, thereby automatically reducing the amount of calculation, speeding up the calculation speed, and well ensuring the high-precision requirements of cutout.

[0049] See also Figure 2 FIG. 1 is a flow chart of an image processing method, which may include:

[0050] Step 201: Divide multiple frames of initial images in an image sequence corresponding to a target object into a first category of initial images to be accelerated and a second category of initial images not to be accelerated based on a target frame interval.

[0051] Exemplarily, the target object is an object that the user is interested in. For example, when the object the user is interested in is a face, the target object is a face; when the object the user is interested in is a human body, the target object is a human body; when the object the user is interested in is a vehicle, the target object is a vehicle. There is no restriction on the type of the target object.

[0052] For example, an image sequence may include multiple frames of initial images, such as RGB images, and each initial image may include a target object. Therefore, the image sequence may be referred to as an image sequence corresponding to the target object. The image sequence may include all initial images in a video file, or may include some initial images in the video file. In other words, all initial images in the video file may be combined into an image sequence, or some initial images in the video file may be combined into an image sequence.

[0053] Exemplarily, the target frame interval represents the interval between two adjacent second-category initial images, and the target frame interval is recorded as n. Then, every n frames is a second-category initial image, that is, the interval between two adjacent second-category initial images is n frames, and all initial images between the two second-category initial images are first-category initial images. Alternatively, the target frame interval represents the interval between two adjacent first-category initial images, and the target frame interval is recorded as m. Then, every m frames is a first-category initial image, that is, the interval between two adjacent first-category initial images is m frames, and all initial images between the two first-category initial images are second-category initial images. Regarding the method of obtaining the target frame interval, please refer to the subsequent embodiments and will not be repeated here.

[0054] For example, assuming the target frame interval represents the interval between two adjacent second-category initial images, if n is 1, then frames 1, 3, 5, ... in the image sequence are second-category initial images, the interval between two adjacent second-category initial images is 1 frame, and all initial images between two adjacent second-category initial images are first-category initial images. That is, frames 2, 4, 6, ... in the image sequence are first-category initial images. If n is 2, then frames 1, 4, 7, ... in the image sequence are second-category initial images, the interval between two adjacent second-category initial images is 2 frames, and all initial images between two adjacent second-category initial images are first-category initial images. That is, frames 2, 3, 5, 6, 8, 9, ... in the image sequence are first-category initial images. If n is 3, then frames 1, 5, 9, ... in the image sequence are second-category initial images, and frames 2, 3, 4, 6, 7, 8, 10, 11, 12, ... in the image sequence are first-category initial images.

[0055] From the above, it can be seen that when the target frame interval n is larger, the usage ratio of the second type of initial images is lower, and the usage ratio of the first type of initial images is higher. Since the first type of initial images need to be accelerated, while the second type of initial images do not need to be accelerated, the acceleration effect is more obvious.

[0056] Step 202: For each initial image in the image sequence, when performing cutout processing on the initial image, determine whether the initial image is a first-class initial image or a second-class initial image. If the initial image is a second-class initial image, proceed to step 203; if the initial image is a first-class initial image, proceed to step 204.

[0057] For example, each initial image in the image sequence may be matted sequentially according to the order of the initial images. For example, the first initial image in the image sequence may be matted first, followed by the second initial image in the image sequence, and so on. When matting each initial image, it is necessary to determine whether the initial image is a first-category initial image or a second-category initial image, and then to use different methods to matte the initial image. The specific matting methods will be described in subsequent steps.

[0058] Step 203 : For the second type of initial image, a foreground image corresponding to the target object is obtained based on the second type of initial image. That is, a foreground image is obtained based on the entire second type of initial image.

[0059] Exemplarily, the process of obtaining the foreground image corresponding to the target object based on the second type of initial image is to cut out the second type of initial image. This embodiment does not limit the cutout method. For example, the target network model can be used to cut out the second type of initial image, or other methods can be used for cutout.

[0060] When using a target network model to perform cropping on the second type of initial image, the target network model can be pre-trained. There are no restrictions on the training process of the target network model, as long as the target network model can achieve the image cropping function. The target network model can be a network model based on a deep learning algorithm or a network model based on a neural network, and there are no restrictions on the structure of the target network model.

[0061] Based on the trained target network model, in step 203, the second type of initial image can be input into the target network model to obtain a foreground region in the second type of initial image, and a foreground image corresponding to the target object is generated based on the foreground region. For example, after the second type of initial image is input into the target network model, the target network model can determine the probability that each pixel in the second type of initial image belongs to the foreground region. Based on the probability of each pixel belonging to the foreground region, the region composed of pixels with a relatively high probability (i.e., the probability of the pixel is greater than a preset probability value) can be used as the foreground region.

[0062] After obtaining the foreground area in the second type of initial image, a foreground image corresponding to the target object can be generated based on the foreground area. For example, the foreground area can be used as the foreground image, or the foreground area can be adjusted and the adjusted foreground area can be used as the foreground image. There is no restriction on this.

[0063] See also Figure 3A As shown, this is an example of the second type of initial image, see Figure 3B , which is an example of a foreground image obtained by performing a cutout operation on the second type of initial image.

[0064] In one possible embodiment, for the second type of initial image, a foreground mask image (i.e., an alpha_matte image) corresponding to the second type of initial image needs to be generated. The method for generating this foreground mask image is not limited. Exemplarily, the pixel values ​​in the foreground mask image corresponding to the background region of the second type of initial image are a first value (e.g., 0), and the pixel values ​​in the foreground mask image corresponding to the foreground region of the second type of initial image are greater than the first value (e.g., 1, or a value between 0 and 1).

[0065] For example, the foreground mask image is also called the foreground transparency image or transparency mask image. The foreground mask image is an image that separates the foreground and background. It is a grayscale image. The grayscale value of each pixel represents the degree to which each pixel of the second type of initial image belongs to the foreground area, that is, alpha. Among them, the foreground mask image technology is mainly used for object extraction, that is, image clipping of the initial image. Figure 3C , which is an example of a foreground mask image obtained by performing cutout on the second type of initial image.

[0066] Step 204: For the first type of initial image, determine a candidate initial image corresponding to the first type of initial image, and based on the foreground mask image corresponding to the candidate initial image, determine the target area corresponding to the target object in the first type of initial image. For example, the previous initial image of the first type of initial image can be used as the candidate initial image, or another initial image preceding the first type of initial image can be used as the candidate initial image. This embodiment does not impose any restrictions on the candidate initial images.

[0067] For example, considering the continuity of the video, when processing each frame of the initial image, the position of the target object in the initial image will not undergo a large leap compared with the previous frame of the initial image, that is, the target object will still be in the vicinity of the area where the previous frame of the initial image is located. Therefore, the previous frame of the initial image of the first type of initial image can be used as a candidate initial image, and the target area corresponding to the target object in the first type of initial image can be determined based on the foreground mask image corresponding to the candidate initial image.

[0068] In a possible implementation, the following steps can be used to determine the target area corresponding to the target object in the first type of initial image. Of course, the following method is just an example and is not limited to this in this embodiment, as long as the target area can be determined based on the foreground mask image.

[0069] Step 2041: Based on the total pixel value of each row of the foreground mask image corresponding to the candidate initial image (the total pixel value is the sum of the pixel values ​​of all pixels in the row), determine the upper boundary and the lower boundary, the total pixel value of the upper boundary is greater than the preset threshold, and the total pixel value of the lower boundary is greater than the preset threshold.

[0070] Step 2042: Based on the total pixel values ​​of each column of the foreground mask image corresponding to the candidate initial image (the total pixel value is the sum of the pixel values ​​of all pixels in the column), determine the left boundary and the right boundary, the total pixel value of the left boundary is greater than the preset threshold, and the total pixel value of the right boundary is greater than the preset threshold.

[0071] For example, since the pixel value corresponding to the background area of ​​the candidate initial image in the foreground mask image is a first value (such as 0), and the pixel value corresponding to the foreground area of ​​the candidate initial image in the foreground mask image is greater than the first value (such as 1, or a value between 0 and 1), for each row of the foreground area, the total pixel value of the row can be relatively large, such as greater than a preset threshold. The preset threshold can be configured based on experience, such as 0, or a value greater than 0, and there is no restriction on this preset threshold. Similarly, for each column of the foreground area, the total pixel value of the column can be relatively large, such as greater than the preset threshold.

[0072] Based on the above principle, in this embodiment, in order to determine the four boundaries of the target area, namely the upper, lower, left and right boundaries, an accelerated algorithm can be used to calculate the total value of the pixels in each row of the foreground mask image and the total value of the pixels in each column of the foreground mask image. The upper and lower boundaries are determined based on the total value of the pixels in each row, and the left and right boundaries are determined based on the total value of the pixels in each column. Taking the total value of the pixels in each column as an example, see Figure 4 As shown, assuming that the width of the foreground mask image is 800, the total pixel value of each column can be obtained, that is, the total pixel value of 800 columns can be obtained. In the [0, 305]th column, the total pixel value of these columns is 0, in the [633, 799]th column, the total pixel value of these columns is 0, and in the [306, 632]th column, the total pixel value of these columns is greater than 0. Therefore, if the preset threshold is 0, it can be determined that the left boundary is the 306th column and the right boundary is the 632nd column. Similarly, the total pixel value of each row can be determined, and the upper and lower boundaries can be determined based on the total pixel value of each row.

[0073] To sum up, the upper and lower boundaries can be determined based on the total value of pixels in each row, and the left and right boundaries can be determined based on the total value of pixels in each column. The total value of pixels of the upper boundary, the total value of pixels of the lower boundary, the total value of pixels of the left boundary, and the total value of pixels of the right boundary all need to be greater than the preset threshold.

[0074] Step 2043 : determining the target region based on the upper boundary, the lower boundary, the left boundary, and the right boundary, that is, the target region corresponding to the target object in the first type of initial image. The target region may be a rectangular region.

[0075] For example, considering the mobility of the target object, in actual use, a certain range will be expanded based on the upper, lower, left, and right boundaries to serve as the target object's possible motion range. Based on this, the upper boundary can be expanded upward by p1 pixel rows. If the expanded pixel rows exceed the first row of the first-category initial image, the first row of the first-category initial image is used as the target upper boundary. If the expanded pixel rows do not exceed the first row of the first-category initial image, the expanded pixel rows are used as the target upper boundary. The lower boundary can be expanded downward by p2 pixel rows. If the expanded pixel rows exceed the last row of the first-category initial image, the last row of the first-category initial image is used as the target lower boundary. If the expanded pixel rows do not exceed the last row of the first-category initial image, the expanded pixel rows are used as the target lower boundary. The left boundary can be expanded left by p3 pixel columns. If the expanded pixel columns exceed the first column of the first-category initial image, the first column of the first-category initial image is used as the target left boundary. If the expanded pixel columns do not exceed the first column of the first-category initial image, the expanded pixel columns are used as the target left boundary. Based on the right boundary, p4 pixel columns can be extended to the right. If the extended pixel columns exceed the last column range of the first type of initial image, the last column of the first type of initial image will be used as the target right boundary. If the extended pixel columns do not exceed the last column range of the first type of initial image, the extended pixel columns will be used as the target right boundary.

[0076] For example, after obtaining the target upper boundary, target lower boundary, target left boundary and target right boundary, the rectangular area formed by the target upper boundary, target lower boundary, target left boundary and target right boundary can be used as the target area corresponding to the target object in the first type of initial image.

[0077] Exemplarily, p1, p2, p3 and p4 can all be configured based on experience, and there is no limitation to this. p1, p2, p3 and p4 can be all the same, partially the same, or different.

[0078] In one possible implementation, the search direction can be accelerated, and the search direction is not from left to right, nor from left to bottom, but from outside to inside, see Figure 5AAs shown in FIG, it is a schematic diagram of searching from the outside to the inside. Once a condition is met (such as the total value of the pixels in the current row (column) is greater than a preset threshold), it is determined to be a boundary. Exemplarily, the search method from the outside to the inside is adopted. The reason is that: assuming that the width and height of the foreground mask image are W and H, this search method often does not need to search all the W*H spaces (unless there is no target object) to obtain the boundary of the target object. In addition, it can avoid finding false boundaries. For example, if there are multiple target objects in the picture, there will be discontinuous areas. The search method from left to right may cause some target objects to be missing, and the search method from top to bottom may cause some target objects to be missing. However, the search method from the outside to the inside will not cause some target objects to be missing.

[0079] For example, when using an outside-in search method, a first target row can be traversed starting from the first row on the upper side of the foreground mask image. The first target row is the row on the upper side where the total pixel value is greater than a preset threshold, and the first target row is determined as the upper boundary. For example, first determine whether the total pixel value of the first row on the upper side is greater than the preset threshold. If so, the first row on the upper side is determined as the first target row. If not, determine whether the total pixel value of the second row on the upper side is greater than the preset threshold. If so, the second row on the upper side is determined as the first target row, and so on. A second target row can be traversed starting from the first row on the lower side of the foreground mask image. The second target row is the row on the lower side where the total pixel value is greater than the preset threshold. The second target row is determined as the lower boundary. For example, first determine whether the total pixel value of the first row on the lower side is greater than the preset threshold. If so, the first row on the lower side is determined as the second target row. If not, determine whether the total pixel value of the second row on the lower side is greater than the preset threshold. If so, the second row on the lower side is determined as the second target row, and so on.

[0080] For example, when using an outside-in search method, a first target column can be traversed starting from the first column on the left side of the foreground mask image. The first target column is the first column on the left side whose total pixel value is greater than a preset threshold. This first target column is determined as the left boundary. For example, first determine whether the total pixel value of the first column on the left side is greater than the preset threshold. If so, the first column on the left side is used as the first target column. If not, determine whether the total pixel value of the second column on the left side is greater than the preset threshold. If so, the second column on the left side is used as the first target column, and so on. A second target column can be traversed starting from the first column on the right side of the foreground mask image. The second target column is the first column on the right side whose total pixel value is greater than the preset threshold. This second target column is determined as the right boundary. For example, first determine whether the total pixel value of the first column on the right side is greater than the preset threshold. If so, the first column on the right side is used as the second target column. If not, determine whether the total pixel value of the second column on the right side is greater than the preset threshold. If so, the second column on the right side is used as the second target column, and so on.

[0081] In one possible implementation, the search mode can be accelerated. Considering that there may be a large background area, especially when the target object occupies a small area, the processing efficiency of row and column by row is relatively low, and the target object generally has a certain size. Therefore, a value can be set for skip search. For example, the skip value is set to skip = 10, that is, each time the detection is performed at intervals of 10 pixels (rows or columns). If the total pixel value is 0, the detection will continue to jump 10 pixels. If the total pixel value is not 0, it will go back 9 pixels and then detect pixel by pixel. That is, the boundary is between these two search positions. By detecting pixel by pixel, the boundary can be accurately obtained.

[0082] From the above, we can see that Figure 5B As shown in the figure, when detecting the left and right boundaries, the large blank areas on the left and right sides are not detected column by column, but are detected every 10 columns. Similarly, when detecting the upper and lower boundaries, the large blank areas on the upper and lower sides are not detected row by row, but are detected every 10 rows.

[0083] Obviously, when the rows and columns are searched with 10-bit jumps, there will be a nearly 100-fold acceleration effect.

[0084] Exemplarily, determine whether the total pixel value of the 1st upper row is greater than 0. If so, use the 1st upper row as the upper boundary. If not, determine whether the total pixel value of the 11th upper row (skip K rows) is greater than 0. If not, determine whether the total pixel value of the 21st upper row (skip K rows) is greater than 0. If yes, determine whether the total pixel value of the 12th upper row (fall back K-1 rows) is greater than 0. If not, determine whether the total pixel value of the 13th upper row is greater than 0. If yes, use the 13th upper row as the upper boundary.

[0085] In summary, the upper boundary can be obtained. Similarly, the lower boundary, left boundary and right boundary can be obtained. The method of obtaining the lower boundary, left boundary and right boundary is similar to that of the upper boundary, and will not be repeated here.

[0086] Step 205: intercept a target sub-image corresponding to the target area from the first type of initial image.

[0087] Exemplarily, since the target region represents the region corresponding to the target object in the first type of initial image, a target sub-image corresponding to the target region can be captured from the first type of initial image.

[0088] Step 206: Acquire a foreground image corresponding to the target object based on the target sub-image.

[0089] Exemplarily, for the first type of initial image, a foreground image corresponding to the target object can be obtained based on the target sub-image in the first type of initial image. Since the target sub-image is a sub-image in the first type of initial image, that is, the size of the target sub-image is smaller than the size of the first type of initial image, the computational cost of clipping the target sub-image is smaller than the computational cost of clipping the first type of initial image, thereby reducing the computational cost.

[0090] Exemplarily, the process of obtaining the foreground image corresponding to the target object based on the target sub-image is to perform cropping on the target sub-image. This embodiment does not limit the cropping method. For example, the target sub-image can be cropped using a target network model, or other methods can be used. The target network model corresponding to the target sub-image and the target network model corresponding to the second type of initial image can be the same or different.

[0091] When the target sub-image is cut out using the target network model, the target sub-image can be input into the target network model to obtain the foreground area in the target sub-image, and a foreground image corresponding to the target object is generated based on the foreground area. For example, after the target sub-image is input into the target network model, the target network model can determine the probability that each pixel in the target sub-image belongs to the foreground area. Based on the probability that each pixel belongs to the foreground area, the area composed of pixels with a relatively high probability (i.e., the probability of the pixel is greater than a preset probability value) can be used as the foreground area. After obtaining the foreground area in the target sub-image, the foreground image corresponding to the target object can be generated based on the foreground area, and there is no restriction on this.

[0092] See also Figure 6A , which is an example of the first type of initial image. The target area is shown in the first type of initial image. A target sub-image corresponding to the target area can be intercepted from the first type of initial image. Figure 6B Then, Figure 6B The target sub-image shown is input to the target network model to obtain the foreground area in the target sub-image, and generate the foreground image corresponding to the target object based on the foreground area, see Figure 6C As shown in the figure, the foreground image is the result of the cutout. Then, the missing background part can be directly set as the background. Figure 6D As shown, Figure 6D This is an image with foreground and background separated. Obviously, the unprocessed background area can be directly processed as the background in the next frame, and the final result is the same as the original image. Since the resolution of the target area is not reduced, the quality of the resulting effect can be guaranteed.

[0093] In a possible implementation, for the first type of initial image, it is also necessary to generate a foreground mask image corresponding to the first type of initial image, such as based on Figure 6B The target sub-image shown generates a foreground mask image. There is no restriction on the generation method of the foreground mask image. Figure 6E The foreground mask image shown is an example of a foreground mask image obtained by cropping a target sub-image. Exemplarily, the pixel values ​​corresponding to the background area of ​​the target sub-image in the foreground mask image are a first value (e.g., 0), and the pixel values ​​corresponding to the foreground area of ​​the target sub-image in the foreground mask image are greater than the first value (e.g., 1).

[0094] In one possible implementation, for step 201, multiple frames of initial images need to be divided into first-category initial images and second-category initial images based on a target frame interval. Regarding how the target frame interval is obtained, a configured initial frame interval can be determined as the target frame interval. For example, if the initial frame interval is 2, then the target frame interval is 2; if the initial frame interval is 3, then the target frame interval is 2, and so on. The initial frame interval can be a frame interval configured based on experience, and there is no restriction on this initial frame interval.

[0095] In practical applications, the larger the target frame interval, the lower the proportion of the second type of initial images used, while the higher the proportion of the first type of initial images used, the more significant the acceleration effect. However, considering that the target object may move rapidly, this may lead to deviations in the extracted target region. If the target frame interval is relatively small, the impact of the deviation in the target region is not significant. However, if the target frame interval is relatively large, the deviation in the target region may cause error accumulation, resulting in the inability to find the accurate target region.

[0096] In view of the above findings, in an embodiment of the present application, the initial frame interval can also be adjusted based on the motion state corresponding to the target object, and the adjusted frame interval can be determined as the target frame interval. For example, the frame interval can be dynamically adjusted (when the frame interval a is currently used, the frame interval a is recorded as the initial frame interval, and the frame interval a is adjusted. When the frame interval b is currently used, the frame interval b is recorded as the initial frame interval, and the frame interval b is adjusted, and so on), that is, the size of the target frame interval is dynamically changed according to the motion state of the target object. For example, when the target object is within a certain range for a long time (such as in a video conference, the target object is generally within a certain range), the size of the target frame interval can be gradually increased. When the target object moves significantly, the size of the target frame interval can be reduced, or even temporarily set to 0. When the target object stabilizes again, the size of the target frame interval can be gradually increased.

[0097] In summary, it can be seen that the motion state corresponding to the target object can be determined. If the motion state indicates that the target object is in a stable state (for example, the target object is within a certain range for a long time), the initial frame interval is increased (the currently used frame interval is recorded as the initial frame interval), and the adjusted frame interval is used as the target frame interval. If the motion state indicates that the target object is in an unstable state (for example, the target object undergoes large movements), the initial frame interval is reduced, and the adjusted frame interval is used as the target frame interval.

[0098] Exemplarily, the initial frame interval can be adjusted multiple times. For example, when determining the motion state corresponding to the target object, if the target object is in a stable state, the initial frame interval is increased, and it is determined whether the adjusted frame interval is greater than the maximum frame interval (n_max). If not, the adjusted frame interval is used as the target frame interval. If so, the maximum frame interval is used as the target frame interval. When determining the motion state corresponding to the target object again, if the target object is still in a stable state, the initial frame interval (i.e., the frame interval after the last adjustment) is continued to be increased, and it is determined whether the adjusted frame interval is greater than the maximum frame interval. If not, the adjusted frame interval is used as the target frame interval. If so, the maximum frame interval is used as the target frame interval, and so on.

[0099] For another example, when determining the motion state corresponding to the target object, if the target object is in an unstable state, the initial frame interval is reduced, and it is determined whether the adjusted frame interval is less than 0. If not, the adjusted frame interval is used as the target frame interval. If so, the target frame interval is set to 0. When determining the motion state corresponding to the target object again, if the target object is still in an unstable state, the initial frame interval (i.e., the last adjusted frame interval) is further reduced, and it is determined whether the adjusted frame interval is less than 0. If not, the adjusted frame interval is used as the target frame interval. If so, the target frame interval is set to 0.

[0100] For example, when increasing the initial frame interval, the value of each increase is set to w1. w1 can be configured based on experience, such as 1, 2, etc., without limitation. That is, w1 is added to the initial frame interval. When decreasing the initial frame interval, the value of each decrease is set to w2. w2 can be configured based on experience, such as 1, 2, etc., without limitation. That is, w2 is subtracted from the initial frame interval.

[0101] In one possible implementation, to determine the motion state corresponding to the target object, the following method may be employed. Of course, the following method is merely an example and is not limiting, as long as the motion state corresponding to the target object can be determined. The motion state corresponding to the target object is determined based on the difference between the target boundaries of M frames of foreground mask images corresponding to the M frames of initial images. M may be a positive integer greater than 1. The M frames of initial images may be initial images preceding the first type of initial image in the image sequence. The target boundary may include at least one of an upper boundary, a lower boundary, a left boundary, and a right boundary.

[0102] For example, assuming that M is 5, when the initial image x is cut out, if the initial image x is a first-type initial image, then the M-frame initial image is the 1st, 2nd, 3rd, 4th, and 5th frame initial image before the initial image x. The motion state of the target object can be determined based on the foreground mask images corresponding to these initial images.

[0103] For example, the target boundary may be an upper boundary, a lower boundary, a left boundary, or a right boundary. For another example, the target boundary may include an upper boundary, a lower boundary, a left boundary, and a right boundary simultaneously. For ease of description, the target boundary is taken as an example in which the target boundary includes an upper boundary, a lower boundary, a left boundary, and a right boundary simultaneously.

[0104] Assuming that the initial images of the M frames are x1, x2, x3, x4, and x5, the upper boundary y11, lower boundary y12, left boundary y13, and right boundary y14 of the foreground mask image y1 corresponding to x1 can be determined, and the upper boundary y21, lower boundary y22, left boundary y23, and right boundary y24 of the foreground mask image y2 corresponding to x2 can be determined, and so on. On this basis, the absolute value of the difference between any two upper boundaries (such as y11 and y21) is calculated, the absolute value of the difference between any two lower boundaries (such as y12 and y22) is calculated, the absolute value of the difference between any two left boundaries (such as y13 and y23) is calculated, and the absolute value of the difference between any two right boundaries (such as y14 and y24) is calculated. If at least one of the above values ​​is greater than the threshold move_thresh, it is determined that the motion state corresponding to the target object is unstable and the initial frame interval can be reduced. If all the above values ​​are not greater than the threshold, it is determined that the motion state corresponding to the target object is stable and the initial frame interval can be increased.

[0105] In summary, a threshold move_thresh can be set. If the range of the upper, lower, left, and right boundaries of the target object in the last 5 frames does not exceed this threshold, it can be determined that the target object is in a relatively stable state, and the initial frame interval will gradually increase. Of course, the target frame interval can have an upper limit value n_max, that is, the target frame interval is increased to the upper limit value n_max at most to avoid the accumulation of errors that may occur over a long period of time. If the range of the upper, lower, left, and right boundaries of the target object in the last 5 frames exceeds this threshold (any range exceeds this threshold), it can be determined that the target object is in an unstable state, the initial frame interval will gradually decrease, or the initial frame interval can be accumulated again from 0.

[0106] As can be seen from the above technical solutions, in the embodiments of the present application, since the target sub-image is a sub-image within the first-class initial image, i.e., the size of the target sub-image is smaller than that of the first-class initial image, the computational effort required to cut out the target sub-image is less than that required to cut out the first-class initial image, thereby reducing the computational effort. When using the above method to cut out multiple frames of the first-class initial image, the computational effort can be significantly reduced, the cutting speed is relatively fast, and the cutting process can be completed in a relatively short time, thus meeting real-time requirements. The system can fully utilize the continuity information between images to determine the target region corresponding to the target object within the first-class initial image, thereby adaptively changing the cutting region, thereby automatically reducing the computational effort and accelerating the calculation speed. This effectively ensures the high-precision cutting out requirements without increasing hardware costs, and can adaptively accelerate the cutting out process to ensure global real-time performance. Even for devices with poor performance, a relatively smooth, high-precision cutting out effect can be achieved. The system can adaptively control the target frame interval based on the motion state to achieve a balance between accuracy and speed. This system can adaptively accelerate the target cutting out without increasing hardware costs or changing the network structure, resulting in a faster intuitive experience.

[0107] Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application, see Figure 7 FIG. 1 is a schematic diagram of the structure of the image processing device, which may include:

[0108] A division module 71 is configured to divide a plurality of initial image frames in an image sequence corresponding to a target object into a first type of initial images to be accelerated and a second type of initial images not to be accelerated based on a target frame interval;

[0109] a determination module 72 configured to determine, for the first type of initial image, a target region corresponding to the target object in the first type of initial image based on a foreground mask image corresponding to a candidate initial image; the candidate initial image being an initial image preceding the first type of initial image in the image sequence;

[0110] The acquisition module 73 is configured to capture a target sub-image corresponding to the target area from the first type of initial image; and acquire a foreground image corresponding to the target object based on the target sub-image.

[0111] Exemplarily, the pixel value corresponding to the background area of ​​the candidate initial image in the foreground mask image is a first value, and the pixel value corresponding to the foreground area of ​​the candidate initial image in the foreground mask image is greater than the first value; when the determination module 72 determines the target area corresponding to the target object in the first type of initial image based on the foreground mask image corresponding to the candidate initial image, it is specifically used to: determine the upper and lower boundaries based on the total pixel values ​​of each row of the foreground mask image, the total pixel value of the upper boundary is greater than a preset threshold, and the total pixel value of the lower boundary is greater than the preset threshold; determine the left and right boundaries based on the total pixel value of each column of the foreground mask image, the total pixel value of the left boundary is greater than the preset threshold, and the total pixel value of the right boundary is greater than the preset threshold; determine the target area corresponding to the target object in the first type of initial image based on the upper boundary, the lower boundary, the left boundary and the right boundary.

[0112] Exemplarily, when determining the upper and lower boundaries based on the total pixel values ​​of each row of the foreground mask image, the determination module 72 is specifically used to: traverse a first target row starting from the first row on the upper side of the foreground mask image, the first target row being the first row on the upper side whose total pixel value is greater than a preset threshold, and determine the first target row as the upper boundary; and traverse a second target row starting from the first row on the lower side of the foreground mask image, the second target row being the first row on the lower side whose total pixel value is greater than a preset threshold, and determine the second target row as the lower boundary; when determining the left and right boundaries based on the total pixel values ​​of each column of the foreground mask image, the determination module 72 is specifically used to: traverse a first target column starting from the first column on the left side of the foreground mask image, the first target column being the first column on the left side whose total pixel value is greater than a preset threshold, and determine the first target column as the left boundary; and traverse a second target column starting from the first column on the right side of the foreground mask image, the second target column being the first column on the right side whose total pixel value is greater than the preset threshold, and determine the second target column as the right boundary.

[0113] Exemplarily, when traversing the first target row from the first row on the upper side of the foreground mask image, the determination module 72 is specifically used to: determine whether the total pixel value of the first row on the upper side is greater than a preset threshold; if so, determine the first row on the upper side as the first target row; if not, skip K rows on the basis of the first row on the upper side to obtain a first initial row, and determine whether the total pixel value of the first initial row is greater than a preset threshold, where K is a positive integer greater than 1; if not, skip K rows on the basis of the first initial row, update the row after skipping K rows to the first initial row, and return to execute the operation of determining whether the total pixel value of the first initial row is greater than the preset threshold; if so, go back K-1 rows on the basis of the first initial row to obtain a second initial row, and determine whether the total pixel value of the second initial row is greater than the preset threshold; if yes, determine the second initial row as the first target row; if not, update the next row of the second initial row to the second initial row, and return to execute the operation of determining whether the total pixel value of the second initial row is greater than the preset threshold.

[0114] Exemplarily, the acquisition module 73 is also used to obtain the target frame interval in the following manner: determining the configured initial frame interval as the target frame interval; or adjusting the initial frame interval based on the motion state corresponding to the target object, and determining the adjusted frame interval as the target frame interval.

[0115] Exemplarily, when the acquisition module 73 adjusts the initial frame interval based on the motion state corresponding to the target object, it is specifically used to: determine the motion state corresponding to the target object based on the difference between the target boundaries of the M-frame foreground mask images corresponding to the M-frame initial images; wherein M is a positive integer greater than 1, the M-frame initial image is the initial image located in front of the first type of initial image in the image sequence, and the target boundary includes at least one of the upper boundary, the lower boundary, the left boundary and the right boundary; if the motion state indicates that the target object is in a stable state, then increase the initial frame interval; if the motion state indicates that the target object is in an unstable state, then reduce the initial frame interval.

[0116] Exemplarily, the acquisition module 73 is also used to acquire the foreground image corresponding to the target object based on the second type of initial image; when the acquisition module 73 acquires the foreground image corresponding to the target object based on the target sub-image, it is specifically used to: input the target sub-image to the target network model to obtain the foreground area in the target sub-image; generate the foreground image corresponding to the target object based on the foreground area; when the acquisition module 73 acquires the foreground image corresponding to the target object based on the second type of initial image, it is specifically used to: input the second type of initial image to the target network model to obtain the foreground area in the second type of initial image; generate the foreground image corresponding to the target object based on the foreground area.

[0117] Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application, see Figure 8 As shown, the image processing device includes a processor 81 and a machine-readable storage medium 82, and the machine-readable storage medium 82 stores machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0118] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0119] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0120] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0121] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0122] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0126] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Dividing multiple frames of initial images in an image sequence corresponding to a target object into first-category initial images to be accelerated and second-category initial images not to be accelerated based on a target frame interval; wherein the target frame interval represents the number of first-category initial frame images included between two adjacent second-category initial images, and obtaining the target frame interval includes: adjusting the initial frame interval based on a motion state corresponding to the target object, and determining the adjusted frame interval as the target frame interval; For the first type of initial image, determining a target area corresponding to the target object in the first type of initial image based on a foreground mask image corresponding to a candidate initial image; wherein the candidate initial image is an initial image located before the first type of initial image in the image sequence; intercepting a target sub-image corresponding to the target area from the first type of initial images; A foreground image corresponding to the target object is acquired based on the target sub-image.

2. The method according to claim 1, characterized in that The pixel value corresponding to the background area of ​​the candidate initial image in the foreground mask image is a first value, and the pixel value corresponding to the foreground area of ​​the candidate initial image in the foreground mask image is greater than the first value; The determining, based on the foreground mask image corresponding to the candidate initial image, a target area corresponding to the target object in the first type of initial image includes: Determining an upper boundary and a lower boundary based on the total pixel values ​​of each row of the foreground mask image, wherein the total pixel value of the upper boundary is greater than a preset threshold, and the total pixel value of the lower boundary is greater than a preset threshold; Determining a left boundary and a right boundary based on the total pixel values ​​of each column of the foreground mask image, wherein the total pixel value of the left boundary is greater than a preset threshold, and the total pixel value of the right boundary is greater than a preset threshold; The target area is determined based on the upper boundary, the lower boundary, the left boundary, and the right boundary.

3. The method according to claim 2, characterized in that The determining of the upper boundary and the lower boundary based on the total value of pixels in each row of the foreground mask image comprises: Traversing a first target row from the first row on the upper side of the foreground mask image, the first target row being the first row on the upper side whose total pixel value is greater than a preset threshold, and determining the first target row as the upper boundary; Traversing a second target row from the first row on the lower side of the foreground mask image, the second target row being the first row on the lower side whose total pixel value is greater than a preset threshold, and determining the second target row as the lower boundary; The determining of the left boundary and the right boundary based on the total pixel values ​​of each column of the foreground mask image comprises: Traversing a first target column starting from the first column on the left side of the foreground mask image, the first target column being the first column on the left side whose total pixel value is greater than a preset threshold, and determining the first target column as the left boundary; A second target column is traversed from the first column on the right side of the foreground mask image. The second target column is the first column on the right side whose total pixel value is greater than a preset threshold. The second target column is determined as the right boundary.

4. The method according to claim 3, characterized in that The step of traversing the first target row from the first row on the upper side of the foreground mask image includes: Determine whether the total pixel value of the first upper row is greater than a preset threshold; if so, determine the first upper row as the first target row; if not, skip K rows based on the first upper row to obtain a first initial row, and determine whether the total pixel value of the first initial row is greater than a preset threshold, where K is greater than 1; If not, skip K rows based on the first initial row, update the row after the K rows are skipped as the first initial row, and return to the operation of determining whether the total pixel value of the first initial row is greater than a preset threshold; If so, then based on the first initial row, go back K-1 rows to obtain the second initial row, and determine whether the total pixel value of the second initial row is greater than the preset threshold; if so, determine the second initial row as the first target row; if not, update the next row of the second initial row to the second initial row, and return to execute the operation of determining whether the total pixel value of the second initial row is greater than the preset threshold.

5. The method according to claim 1, wherein The adjusting the initial frame interval based on the motion state corresponding to the target object includes: determining a motion state corresponding to the target object based on a difference between target boundaries of M frames of foreground mask images corresponding to M frames of initial images, wherein M is a positive integer greater than 1, the M frames of initial images are initial images preceding the first type of initial images in the image sequence, and the target boundaries include at least one of an upper boundary, a lower boundary, a left boundary, and a right boundary; If the motion state indicates that the target object is in a stable state, the initial frame interval is increased; if the motion state indicates that the target object is in an unstable state, the initial frame interval is decreased.

6. The method according to claim 1, characterized in that After dividing the plurality of initial images in the image sequence corresponding to the target object into a first category of initial images to be accelerated and a second category of initial images not to be accelerated based on the target frame interval, the method further comprises: acquiring a foreground image corresponding to the target object based on the second category of initial images; Wherein, obtaining the foreground image corresponding to the target object based on the target sub-image includes: inputting the target sub-image into a target network model to obtain a foreground area in the target sub-image; generating the foreground image corresponding to the target object based on the foreground area; Among them, obtaining the foreground image corresponding to the target object based on the second type of initial image includes: inputting the second type of initial image into the target network model to obtain the foreground area in the second type of initial image; and generating the foreground image corresponding to the target object based on the foreground area.

7. An image processing device, characterized in that The device comprises: a division module configured to divide, based on a target frame interval, a plurality of initial image frames in an image sequence corresponding to a target object into a first category of initial images to be accelerated and a second category of initial images not to be accelerated; wherein the target frame interval represents the number of first category initial frame images included between two adjacent second category initial images, and a method for obtaining the target frame interval includes: adjusting the initial frame interval based on a motion state corresponding to the target object, and determining the adjusted frame interval as the target frame interval; a determination module configured to determine, for the first type of initial image, a target region corresponding to the target object in the first type of initial image based on a foreground mask image corresponding to a candidate initial image; the candidate initial image being an initial image preceding the first type of initial image in the image sequence; An acquisition module is configured to intercept a target sub-image corresponding to the target area from the first type of initial image; and acquire a foreground image corresponding to the target object based on the target sub-image.

8. The device according to claim 7, It is characterized in that in, The pixel value corresponding to the background area of ​​the candidate initial image in the foreground mask image is a first value, and the pixel value corresponding to the foreground area of ​​the candidate initial image in the foreground mask image is greater than the first value; when the determination module determines the target area corresponding to the target object in the first type of initial image based on the foreground mask image corresponding to the candidate initial image, it is specifically used to: determine the upper boundary and the lower boundary based on the total pixel value of each row of the foreground mask image, the total pixel value of the upper boundary is greater than a preset threshold, and the total pixel value of the lower boundary is greater than the preset threshold; determine the left boundary and the right boundary based on the total pixel value of each column of the foreground mask image, the total pixel value of the left boundary is greater than the preset threshold, and the total pixel value of the right boundary is greater than the preset threshold; Determine a target area corresponding to the target object in the first type of initial image based on the upper boundary, the lower boundary, the left boundary, and the right boundary; wherein, when determining the upper and lower boundaries based on the total pixel values ​​of each row of the foreground mask image, the determination module is specifically configured to: traverse a first target row starting from the first row on the upper side of the foreground mask image, the first target row being the first row on the upper side whose total pixel value is greater than a preset threshold, and determine the first target row as the upper boundary; and traverse a second target row starting from the first row on the lower side of the foreground mask image, the second target row being the first row on the lower side whose total pixel value is greater than a preset threshold, and determine the second target row as the lower boundary; when determining the left and right boundaries based on the total pixel values ​​of each column of the foreground mask image, the determination module is specifically configured to: traverse a first target column starting from the first column on the left side of the foreground mask image, the first target column being the first column on the left side whose total pixel value is greater than a preset threshold, and determine the first target column as the left boundary; and traverse a second target column starting from the first column on the right side of the foreground mask image, the second target column being the first column on the right side whose total pixel value is greater than the preset threshold, and determine the second target column as the right boundary; Wherein, when the determination module traverses the first target row starting from the first row on the upper side of the foreground mask image, it is specifically used to: determine whether the total pixel value of the first row on the upper side is greater than a preset threshold; if so, determine the first row on the upper side as the first target row; if not, skip K rows on the basis of the first row on the upper side to obtain a first initial row, and determine whether the total pixel value of the first initial row is greater than a preset threshold, where K is a positive integer greater than 1; if not, skip K rows on the basis of the first initial row, update the row after skipping K rows as the first initial row, and return to execute the operation of determining whether the total pixel value of the first initial row is greater than the preset threshold; if so, go back K-1 rows on the basis of the first initial row to obtain a second initial row, and determine whether the total pixel value of the second initial row is greater than the preset threshold; if yes, determine the second initial row as the first target row; if not, update the next row of the second initial row as the second initial row, and return to execute the operation of determining whether the total pixel value of the second initial row is greater than the preset threshold; The acquisition module, when adjusting the initial frame interval based on the motion state corresponding to the target object, is specifically configured to: determine the motion state corresponding to the target object based on a difference between target boundaries of M frames of foreground mask images corresponding to M frames of initial images; wherein M is a positive integer greater than 1, the M frames of initial images are initial images located before the first type of initial images in the image sequence, and the target boundaries include at least one of an upper boundary, a lower boundary, a left boundary, and a right boundary; if the motion state indicates that the target object is in a stable state, increase the initial frame interval; if the motion state indicates that the target object is in an unstable state, decrease the initial frame interval; Among them, the acquisition module is also used to obtain the foreground image corresponding to the target object based on the second type of initial image; when the acquisition module obtains the foreground image corresponding to the target object based on the target sub-image, it is specifically used to: input the target sub-image to the target network model to obtain the foreground area in the target sub-image; generate the foreground image corresponding to the target object based on the foreground area; when the acquisition module obtains the foreground image corresponding to the target object based on the second type of initial image, it is specifically used to: input the second type of initial image to the target network model to obtain the foreground area in the second type of initial image; generate the foreground image corresponding to the target object based on the foreground area.

9. An image processing device, characterized in that include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps described in any one of claims 1-6.

Images