Moving target extraction method, device, computer equipment and storage medium

Abstract

The present invention belongs to the field of video processing, and specifically provides a method, apparatus, computer equipment and storage medium for extracting a moving target, aiming to solve the problem of how to segment a moving target from an image and accurately obtain the contour of the moving target and the motion trajectory of the moving target. To this end, the method of the present invention includes: selecting a video image by skipping frames; sending the image to be processed and the background image into a trained foreground extraction network for semantic segmentation, thereby obtaining a mask image containing the foreground target; and fusing the information of multiple mask images to extract the moving target. Applying the method of the present invention, the speed of moving target recognition is improved by adopting a semantic segmentation-based convolutional neural network; the video processing speed is further accelerated by the frame skipping strategy; and by sending the image to be processed and the background image into the convolutional neural network at the same time, the accuracy of foreground target segmentation is improved while greatly improving the anti-interference ability of the network.

Description

Technical Field

[0001] The present invention belongs to the field of video processing, and specifically provides a moving target extraction method, device, computer equipment and storage medium. Background Art

[0002] Video condensation is a simple summary of video content. It extracts moving objects from the video through algorithmic analysis. Then, the motion trajectories of each object are analyzed, and the different objects are stitched together into a common background scene. These objects are then combined in a certain way to simultaneously display multiple objects and activities occurring at different times. This allows viewers to watch several or even dozens of hours of video in just a few minutes. Extracting moving objects is a critical step in video condensation, as it directly affects the quality and accuracy of the resulting composite video.

[0003] Current moving object extraction faces the following major challenges. First, the category of moving objects is uncertain. Some common object extraction algorithms, such as object detection or instance segmentation, can only extract objects of a fixed category. Their recall rate for categories not present in their training set is very low, making them less versatile for video enrichment. Second, for objects that remain relatively stationary for a certain period of time, some algorithms can extract moving objects in videos by analyzing information from multiple frames before and after. However, this often results in low accuracy, and the need to analyze multiple frames results in slow video processing. Third, for subtle perturbations in the video, traditional image processing algorithms, such as frame difference methods, can detect moving objects, but with low accuracy and prone to false detection. While optical flow methods offer high accuracy, they suffer from slow video processing speeds, making them difficult to meet the requirements of real-time detection. Therefore, segmenting moving objects from background images and accurately determining their outline and trajectory has become a pressing issue.

[0004] Accordingly, this field requires a new solution to solve the above problems. Summary of the Invention

[0005] The present invention aims to solve the above technical problem, namely, how to segment a moving target from a background image and accurately obtain the contour of the moving target and the motion trajectory of the moving target.

[0006] In a first aspect, the present invention provides a moving target extraction method, the method comprising:

[0007] Get the background image;

[0008] Acquire a first image from a first video to be processed according to a first frame skipping rule;

[0009] Obtaining a first mask image sequence based on the first image and the background image;

[0010] Acquire a second video to be processed from the first video to be processed according to the first mask image sequence;

[0011] Acquire a second image from the second to-be-processed video according to a second frame skipping rule;

[0012] obtaining a second mask image sequence based on the second image and the background image;

[0013] Based on the second mask image sequence, a motion trajectory of the moving target is acquired.

[0014] In one embodiment of the above-mentioned moving object extraction method, the step of "obtaining a first mask image sequence based on the first image and the background image" specifically includes sequentially inputting the first image and the background image into a trained foreground extraction network for semantic segmentation to obtain the first mask image sequence, wherein the first mask image sequence includes a plurality of first mask images;

[0015] The step of “obtaining a second mask image sequence based on the second image and the background image” specifically includes sequentially inputting the second image and the background image into the trained foreground extraction network for semantic segmentation to obtain the second mask image sequence, where the second mask image sequence includes a plurality of second mask images;

[0016] The foreground extraction network is a convolutional neural network;

[0017] The first mask image and the second mask image have the same semantic information, and the semantic information includes that a position where the pixel value is 1 indicates that a foreground object exists, and a position where the pixel value is 0 indicates that the foreground object does not exist.

[0018] In one embodiment of the above-mentioned moving target extraction method, the method further includes:

[0019] Marking the connected domains with pixel values ​​of 1 in the second mask image;

[0020] According to the connected domain, position information of the foreground object in the second mask image is obtained, where the position information includes a foreground object ID and a rectangular box corresponding to the foreground object ID. The number of the foreground object IDs is one or more.

[0021] In one embodiment of the above-mentioned moving target extraction method, the foreground target includes a first foreground target and a second foreground target, the first foreground target is the foreground target in the first result image, the second foreground target is the foreground target in the second result image, the first result image and the second result image are two adjacent second mask images in the second mask image sequence, and the second result image is the second mask image of the previous frame of the first result image. The position information includes first position information, second position information, and predicted position information, and the first position information, the second position information, and the predicted position information each include the respective foreground target IDs and rectangular boxes corresponding to the respective foreground target IDs;

[0022] The step of “obtaining the motion trajectory of the moving target based on the second mask image sequence” specifically includes:

[0023] Acquiring the first position information of the first foreground object;

[0024] Obtaining the predicted position information of the second foreground object at the moment corresponding to the first result image;

[0025] Obtaining an IoU value between the rectangular box in the first position information and the rectangular box in the predicted position information;

[0026] The motion trajectory of the moving target is obtained according to the IoU value.

[0027] In one embodiment of the above-mentioned moving target extraction method, the step of “obtaining the motion trajectory of the moving target according to the IoU value” specifically includes:

[0028] Acquiring the second position information of the second foreground object;

[0029] Using the IoU value as a weight of a KM algorithm, and obtaining a moving object matching degree between the first foreground object and the second foreground object through the KM algorithm;

[0030] When the moving object matching degree is greater than or equal to a moving object matching degree threshold, it is determined that the first foreground object and the second foreground object are the same moving object;

[0031] The motion trajectory of the moving target is acquired according to the first position information and the second position information.

[0032] In one embodiment of the above-mentioned moving object extraction method, the foreground object further includes a third foreground object, the third foreground object is the foreground object in the third result image, and the third result image is the second mask image of the first M frames of the second result image in the second mask image sequence, where M is an integer greater than or equal to 1;

[0033] The step of “obtaining the predicted position information of the second foreground object at the time corresponding to the first result image” specifically includes:

[0034] Obtaining the current speed of the second foreground object;

[0035] Acquiring a historical speed of the third foreground object, wherein the third foreground object and the second foreground object correspond to the same moving object;

[0036] Obtaining a predicted speed of the second foreground object based on the current speed and the historical speed;

[0037] Obtaining a center point of a rectangular frame of the predicted position information according to the predicted speed and a time difference between the second result image and the first result image;

[0038] The predicted position information is obtained according to the center point of the rectangular frame of the predicted position information.

[0039] In one embodiment of the above-mentioned moving target extraction method, the step of “obtaining the predicted position information according to the center point of the rectangular frame of the predicted position information” specifically includes:

[0040] The width and height of the rectangular box of the predicted position information are respectively the average values ​​of the width and height of the rectangular boxes in the position information of each foreground target corresponding to the same moving target in the second result image and the second mask image N frames before the second result image, where N is an integer greater than or equal to 1.

[0041] In one embodiment of the above-mentioned moving target extraction method, the method of "obtaining a background image" includes:

[0042] Obtaining the initial background image;

[0043] The step of “obtaining the initial background image” specifically includes:

[0044] acquiring a third image from a time range specified within the initial stage of the first video to be processed according to a third frame skipping rule;

[0045] Based on the third image, the initial background image is obtained by a median filtering algorithm.

[0046] In one embodiment of the above-mentioned moving target extraction method, the method of "obtaining a background image" further includes:

[0047] maintaining the background image;

[0048] The step of “maintaining the background image” specifically includes:

[0049] Acquire a fourth image from the first video to be processed according to a fourth frame skipping rule;

[0050] Sending the fourth image and the historical background image to the trained foreground extraction network to obtain a third mask image, wherein the historical background image is the background image before the time corresponding to the fourth image;

[0051] The background image is updated according to the fourth image, the third mask image and the historical background image.

[0052] In one embodiment of the above-mentioned moving object extraction method, in the process of acquiring the first mask image sequence, the background image is maintained;

[0053] And / or, in the process of acquiring the second mask image sequence, the background image is maintained.

[0054] In one embodiment of the above-mentioned moving target extraction method, the step of “obtaining a second video to be processed from the first video to be processed according to the first mask image sequence” specifically includes:

[0055] Obtaining a foreground object ratio value of the first mask image according to the number of pixels whose pixels are 1 in the first mask image;

[0056] When the foreground object ratio value is greater than or equal to the foreground object ratio threshold, it is determined that the foreground object exists in the first mask image;

[0057] checking one by one whether the first mask images in the first mask image sequence contain the foreground object;

[0058] When the foreground object exists in two adjacent first mask images, the first video to be processed within a time range corresponding to the two adjacent first mask images is acquired to obtain the second video to be processed.

[0059] In a second aspect, the present invention provides a moving target extraction device, comprising:

[0060] A background acquisition module is configured to perform the following operations:

[0061] Get the initial background image,

[0062] maintaining the background image;

[0063] An image acquisition module, wherein the image acquisition module is configured to perform the following operations:

[0064] According to the first frame skipping rule, a first image is obtained from the first video to be processed.

[0065] According to the first mask image sequence, a second video to be processed is obtained from the first video to be processed,

[0066] Acquire a second image from the second to-be-processed video according to a second frame skipping rule;

[0067] A foreground object segmentation module is configured to perform the following operations:

[0068] Based on the first image and the background image, a first mask image sequence is obtained,

[0069] obtaining a second mask image sequence based on the second image and the background image;

[0070] A moving target extraction module is configured to obtain a motion trajectory of the moving target according to the second mask image sequence.

[0071] In one embodiment of the above-mentioned moving object extraction device, the foreground object segmentation module is configured to perform the following specific operations:

[0072] sequentially inputting the first image and the background image into a trained foreground extraction network for semantic segmentation to obtain a first mask image sequence, where the first mask image sequence includes multiple first mask images;

[0073] sequentially sending the second image and the background image to the trained foreground extraction network for semantic segmentation to obtain a second mask image sequence, where the second mask image sequence includes multiple second mask images;

[0074] The foreground extraction network is a convolutional neural network;

[0075] The first mask image and the second mask image have the same semantic information, and the semantic information includes that a position where the pixel value is 1 indicates that a foreground object exists, and a position where the pixel value is 0 indicates that the foreground object does not exist.

[0076] In a third aspect, the present invention proposes a computer device comprising a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and is characterized in that the program code is suitable for being loaded and run by the processor to execute the motion target extraction method described in any of the above-mentioned schemes.

[0077] In a fourth aspect, the present invention proposes a storage medium, which is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by a processor to execute the motion target extraction method described in any of the above solutions.

[0078] By employing the above-mentioned technical solution, the present invention uses a semantic segmentation convolutional neural network to improve the speed of moving target recognition and further accelerate video processing through a frame-skipping strategy. Furthermore, by simultaneously feeding the current frame image and the background image into the convolutional neural network, it is possible to segment any class of objects that are not present in the background. Furthermore, the neural network's powerful learning ability can eliminate even minor disturbances in the video, improving the network's anti-interference capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:

[0080] Figure 1 This is a flow chart of the main steps of the moving target extraction method according to an embodiment of the present invention.

[0081] Figure 2 Schematic diagram of semantic segmentation of a foreground extraction network according to an embodiment of the present invention.

[0082] Figure 3 yes Figure 1 Specific implementation flow chart of step S107 in .

[0083] Figure 4 yes Figure 3 Specific implementation flow chart of step S1072 in .

[0084] Figure 5 It is a schematic diagram of the composition structure of the moving target extraction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0085] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0086] It should be understood by those skilled in the art that these embodiments are merely used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art may make adjustments as needed to adapt to specific applications.

[0087] Read First Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of the moving target extraction method according to an embodiment of the present invention. Figure 1 As shown, the moving target extraction method of the embodiment of the present invention includes:

[0088] Step S101: obtaining a background image;

[0089] Step S102: acquiring a first image from a first video to be processed according to a first frame skipping rule;

[0090] Step S103: obtaining a first mask image sequence based on the first image and the background image;

[0091] Step S104: obtaining a second video to be processed from the first video to be processed according to the first mask image sequence;

[0092] Step S105: acquiring a second image from a second video to be processed according to a second frame skipping rule;

[0093] Step S106: obtaining a second mask image sequence based on the second image and the background image;

[0094] Step S107: obtaining the motion trajectory of the moving target based on the second mask image sequence.

[0095] In this embodiment, the video acquisition device is a camera with a fixed position and angle, such as a security camera in a supermarket or a security camera on a road. The scene captured by the camera is relatively fixed within a certain time range, that is, the video has a relatively fixed background image.

[0096] When it is necessary to condense historical videos within a certain time period, in step S101, it is first necessary to obtain the initial background image of the working scene of the video to be processed. Preferably, according to the third frame skipping rule, images within a specified time range at the beginning of the first video to be processed are intercepted to obtain a third image, wherein the first video to be processed is the video file to be condensed. As an example, the third frame skipping rule can be set to intercept one frame of image every 100 frames, and the specified time range is the first 30 seconds of video in the video file. When the first video to be processed includes 25 frames per second, images in the first video to be processed are intercepted starting from the first frame of image to obtain 8 third images of the same size.

[0097] Preferably, the eight third images are fused using a median filter algorithm to obtain an initial background image. As an example, when the third images are in RGB format, the color data of the pixels at the same position in the eight third images can be considered as three groups (R, G, B) of one-dimensional sequences with eight data points. A one-dimensional median filter is then performed on the eight R values, eight G values, and eight B values ​​of the pixels at the same position in the eight third images, respectively, to obtain an initial background image that better reflects the actual background and incorporates information from the eight third images.

[0098] In another embodiment, the eight third images are first subjected to a two-dimensional median filter process using a matrix median filter. The size and / or shape of the submatrix window for the two-dimensional median filter can be selected based on practical needs. For example, a 3*3 rectangular window can be selected. Then, according to the aforementioned method, pixels at the same position in the eight third images subjected to the two-dimensional median filter process are subjected to a one-dimensional median filter process to obtain an initial background image. The one-dimensional or two-dimensional median filter algorithm can be implemented using the C language, the OpenCV software library, or other computer tools.

[0099] In this embodiment, in order to speed up the video processing speed, a frame skipping method is used to detect whether there is a foreground target in the first video to be processed. In step S102, according to the first frame skipping rule, the first image is obtained from the first video to be processed. The more frames skipped by the first frame skipping rule, the faster the detection speed. However, when the interval is too large, the probability of missed detection will greatly increase. Therefore, it is necessary to reasonably select the first frame skipping rule to take into account both efficiency and effect. As an example, when the video file is compressed using the H.264 standard, the first frame skipping rule can be set to intercept the I frame of the decoded video. The reason why I frame is selected as the first frame skipping rule is that I frame is also called key frame. It is an important frame in inter-frame compression coding. It is a full-frame compressed coding frame. During decoding, the complete image can be reconstructed using only the data of I frame.

[0100] In step S103, the first image and background image are fed simultaneously into a trained foreground extraction network in chronological order, performing semantic segmentation. This generates a first mask image sequence consisting of multiple first mask images. Preferably, the foreground extraction network uses the BiSeNet v2 convolutional neural network. BiSeNet v2 is a real-time semantic segmentation network that separates spatial details from classification semantics, achieving high-precision and efficient real-time semantic segmentation.

[0101] like Figure 2The output of the BiSeNet v2 network is a binarized mask image. In this embodiment, the semantic information of the mask image is as follows: positions with a pixel value of 1 (the white portion of the mask image) indicate the presence of a foreground target, and positions with a pixel value of 0 (the black portion of the mask image) indicate the absence of a foreground target and are part of the background image. That is, areas with a pixel value of 1 in the mask image are areas in the image to be processed that may be moving targets. The mask images in this embodiment include the first mask image in step S103, the second mask image in step S106, and the third mask image.

[0102] In step S104, the ratio of the number of pixels with a pixel value of 1 in each first mask image in the first mask image sequence is calculated to obtain a foreground target ratio value. When the foreground target ratio value is greater than or equal to a preset foreground target ratio threshold, it is determined that a foreground target exists in the first mask image. As an example, the foreground target ratio threshold can be set based on the type of moving targets typically appearing in the scene, the size of the moving targets within the camera's field of view, etc. For example, the foreground target ratio threshold can be set to 5%. When the ratio of pixels with a pixel value of 1 in the first mask image exceeds 5%, it is determined that the first mask image contains a foreground target.

[0103] Check one by one whether two adjacent first mask images in the first mask image sequence both contain the foreground object. If both adjacent first mask images contain the foreground object, select the first to-be-processed video within the time range corresponding to the two first mask images as the second to-be-processed video.

[0104] From the above method of obtaining the second video to be processed, it can be seen that the second video to be processed is a video containing a foreground target. In this way, in subsequent data processing, only the first video to be processed containing the foreground target needs to be processed, and there is no need to process the video to be processed with only the background, which greatly improves the speed of image processing.

[0105] In step S105, a second image is obtained from the second video to be processed according to the second frame skipping rule. To more accurately extract moving objects, the second frame skipping rule can be set to be frame-by-frame. In other embodiments, to achieve faster processing speed, the second frame skipping rule can also be set to be every other frame or multiple frames, but this will reduce the accuracy of moving object extraction.

[0106] In step S106, similarly, the second image and background image acquired in step S105 are fed simultaneously into the same trained foreground extraction network as in step S103, sequentially following the video's chronological order, for semantic segmentation. This yields a second mask sequence consisting of multiple second mask images. Similarly, the second mask image is a binary mask image, where a pixel value of 1 indicates the presence of a foreground object, and a pixel value of 0 indicates the absence of a foreground object.

[0107] Before executing step S107, it is necessary to obtain the position information of each second foreground object in the second mask image. First, the connected components in the second mask image with a pixel value of 1 are marked. The method for marking connected components is not limited in this invention. As an example, it can be implemented using the connectedComponentsWithStats function in OpenCV. This function can distinguish different connected components, thereby distinguishing different foreground objects.

[0108] Then, based on the connected domain, a rectangular frame of the connected domain in the second mask image is drawn. The present invention does not limit the method of drawing the rectangular frame. As an example, the rectangular outline of the connected domain can be obtained by the BoundingRect function in OpenCV.

[0109] When there are multiple moving targets in the video, the second mask image will contain multiple rectangular frames. Each rectangular frame is numbered one by one to obtain the position information of the foreground target in the second mask image. The position information includes the foreground target ID and the rectangular frame corresponding to the foreground target ID. The foreground target ID can be one or more. In this embodiment, the upper left vertex of the second mask image is selected as the origin of the pixel coordinate system. The rectangular frame can be represented as S n (u, v, w, h), where u and v are the row and column coordinates of the center point of the rectangle in the pixel coordinate system, w is the width of the rectangle, h is the height of the rectangle, and n is the ID number of the foreground target.

[0110] It should be noted that in step S104 of this embodiment, only the number of 0 and 1 pixels in the first mask image is counted. The presence of a foreground object is determined based on the number of 0 and 1 pixels in the first mask image, and no connected domain or rectangular frame processing is performed. This can further improve video processing speed. Similarly, step S104 can also refer to the method in step S107 to mark the connected domain and obtain the rectangular frame. The proportion of the connected domain or rectangular frame can be used to determine whether a foreground object is present in the first mask image.

[0111] Next, combine Figure 3 , explaining the specific implementation method of step S107.

[0112] According to the time sequence, two adjacent second mask images are selected, namely the first result image and the second result image, and the second result image is the second mask image of the previous frame of the first result image. The first result image corresponds to time A, and the second result image corresponds to time B.

[0113] It should be noted that in step S107, the foreground object includes a first foreground object and a second foreground object. The first foreground object is the foreground object in the first result image, and the second foreground object is the foreground object in the second result image. In this embodiment, the position information for different images or for different purposes further includes first position information, second position information, third position information, predicted position information, etc., all of which share a common technical feature, namely, they all include their respective foreground object IDs and rectangular boxes corresponding to their respective foreground object IDs.

[0114] In step S1071, the first position information of the first foreground object in the first result image is obtained. The first result image corresponds to time A. The first result image contains n first foreground objects. The first position information of the first foreground object can be expressed as

[0115] S(A) n (u1,v1,w1,h1).

[0116] In step S1072, the predicted position information of the second foreground object in the second result image at the time corresponding to the first result image is obtained. The content of the predicted position information is: based on the second result image and the position information of the second mask image within a specified time range before the time B corresponding to the second result image, the predicted position information of the second foreground object at the time A in the second mask image is predicted. The predicted position information can be expressed as S′(A) n (u′,v′,w′,h′).

[0117] Next, combine Figure 4 The specific implementation method of step S1072 is described below. In step S401, the current speed of the second foreground object is obtained. For this purpose, information about the third result image is also required. The third result image is the second mask image in the second mask image sequence that is M frames before the second result image. The third result image corresponds to time C, where M is an integer greater than or equal to 1. For example, M can be set to 1.

[0118] The foreground target also includes a third foreground target, which is the foreground target in the third result image. As can be seen from the above, the position information also includes second position information and third position information. The second position information is the position information of the second foreground target, and the third position information is the position information of the third foreground target. The second position information can be expressed as S(B) n(u2,v2,w2,h2), the third position information can be expressed as S(C) n (u3,v3,w3,h3).

[0119] The position difference between the center point (u2, v2) of the rectangular box in the second position information and the center point (u3, v3) of the rectangular box in the third position information on the U axis and V axis is divided by the time difference between the second result image and the third result image to obtain the second foreground object.

[0120] The current speed on the U axis is: vU(B) n =(u2-u3) / |BC|,

[0121] The current speed on the V axis is: vV(B) n =(v2-v3) / |BC|,

[0122] The positive or negative sign of the velocity indicates the moving direction of the moving target on the U axis or V axis.

[0123] In the same way, based on the information of the second mask image of the Wth frame before the third result image, W is an integer greater than or equal to 1. In step S402, the historical speed of the third foreground object is obtained to obtain the historical speed vU(C) of the third foreground object on the U axis. n and the historical speed of the V axis vV(C) n , as an example, W can be set to 10.

[0124] In step S403, the current speed and the historical speed are combined to obtain the predicted speed of the second foreground object. As an example, the method for obtaining the predicted speed of the U axis is:

[0125] v′U(B) n =α*vU(B) n +(1-α)*vU(C) n ;

[0126] The method for obtaining the predicted speed of the V axis is:

[0127] v′V(B) n =β*vV(B) n +(1-β)*vV(C) n ;

[0128] Here, α and β are pre-set coefficients between 0 and 1. For example, α=0.1 and β=0.1.

[0129] After obtaining the predicted speed, in step S404, the predicted center point of the rectangular frame of the predicted position information is obtained based on the time difference between the second result image and the first result image, and the position information of the second foreground object in the second result image. The specific method is:

[0130] u′=u2+v′U(B) n *|AB|,

[0131] v′=v2+v′V(B) n *|AB|.

[0132] In step S405, after obtaining the predicted center point of the rectangular frame of the predicted position information, the length and width of the rectangular frame in the position information of each foreground object corresponding to the moving object in the second result image and the second mask image N frames before the second result image are averaged to obtain the values ​​of w′ and h′, thereby obtaining the predicted position information S′(A). n (u′,v′,w′,h′).

[0133] Continuing with step S1073, the Intersection-over-Union (IoU) value between the rectangular box in the first position information and the rectangular box in the predicted position information is calculated. The present invention does not limit the method for calculating IoU. As an example, IoU can be calculated using the Rect function of OpevCV.

[0134] In step S1074, the IoU value obtained in step S1073 is used as a weight for the KM algorithm to determine the moving object matching between the first foreground object in the first result image and the second foreground object in the second result image. The KM algorithm is a weighted bipartite graph best matching algorithm. The KM algorithm is well known in the art, and its specific implementation is not further described here.

[0135] When the motion target matching degree is greater than or equal to the motion target matching degree threshold, the first foreground target and the second foreground target are determined to be the same motion target. The foreground target ID of the first foreground target and the foreground target ID of the second foreground target are mapped to the same motion target (the motion target has a unique motion target ID). The motion target ID and the center point position of the rectangular box in each second mask image are recorded in the database, thereby obtaining the corresponding relationship data of the motion target ID, the position in the video to be processed, and the position of the motion target in the image. As an example, the motion target matching degree threshold can be set to 0.3, or it can be set to other values ​​according to actual conditions.

[0136] In chronological order, the images of adjacent frames in the second mask image are matched according to the above method, and the moving target ID of each moving target and its position in each frame image in the video to be processed are recorded, so as to extract all moving targets in the video to be processed.

[0137] It should be noted that the KM algorithm can achieve multi-target matching, that is, when there are multiple first foreground targets and multiple second foreground targets, the KM algorithm can be used to extract the motion trajectories of the multiple targets.

[0138] In order to solve the problem of discontinuous foreground targets due to occlusion or partial frame omission, it can be set that when the second foreground target does not have a matching first foreground target in the first result image, the second foreground target will also be retained for a certain period of time and continue to participate in the matching of the moving target. As an example, the maximum number of subsequent frames retained by the second foreground target is K frames, and the value of K can be set to 50 frames. That is to say, within the subsequent maximum 50 frames, the second foreground target will participate in the matching. If it can be matched successfully within the subsequent 50 frames, it is still considered to be the same moving target, and the trajectory of the moving target is recorded; if it has not been matched successfully for more than 50 frames, the trajectory recording of the moving target corresponding to the second foreground target is terminated. In this way, even if occlusion or partial frame omission occurs, the method of the present invention can continue to track the moving target without the motion trajectory being divided into two segments.

[0139] During foreground object segmentation, the initial background image at the beginning of the video being processed, obtained using a median filter algorithm, is first fed into the foreground extraction network. To obtain a more accurate background image throughout the video extraction process, it is typically necessary to periodically update the background image. Users can choose to update the background image only during the acquisition of the first mask sequence or only during the acquisition of the second mask sequence. Alternatively, the background image can be updated during both the first and second mask sequences, resulting in more accurate object segmentation.

[0140] Next, we will describe the method for maintaining the background image. According to the fourth frame skipping rule, a fourth image is first obtained from the first video to be processed. The fourth image and the historical background image are then fed into the same trained foreground extraction network as in step S103 to obtain a third mask image. The third mask image has the same semantic information as the first and second mask images: similarly, positions with a pixel value of 1 indicate the presence of a foreground object, and positions with a pixel value of 0 indicate the absence of a foreground object. The historical background image is the background image before the time corresponding to the fourth image.

[0141] The fourth image, the third mask image and the historical background image are fused to obtain a new background image. The specific method is as follows: if the pixel value of the position corresponding to the fourth image in the third mask image is 1 (there is a moving target), then the pixel value at this position of the fourth image does not participate in the fusion of the background image, and the pixel value at this position is multiplied by 0; if the pixel value of the third mask image is 0 (there is no moving target), then the pixel value at this position of the fourth image is multiplied by the coefficient θ to obtain the first fused background image.

[0142] If the pixel value of the position corresponding to the historical background image in the third mask image is 1 (there is a moving target), the historical background image at that position does not participate in the fusion of the background image, and the pixel value at that position is multiplied by 0; if the pixel value of the third mask image is 0 (there is no moving target), the pixel value at that position of the historical background image is multiplied by the coefficient 1-θ to obtain the second fused background image.

[0143] The pixel values ​​at corresponding positions in the first fused background image and the second fused background image are added together to obtain a new background image, and the historical background image is updated with the new background image. θ is a pre-set coefficient between 0 and 1. As an example, θ can be set to 0.5.

[0144] It should be noted that the maintenance of the background image can also be combined with whether there is a foreground target in the first mask image of the first mask image sequence to determine the fourth frame skipping rule. Try to select video frames without foreground targets to update the background image to obtain a better background image, which will help improve the accuracy of semantic segmentation.

[0145] It should be noted that the background image, first image, second image, and third image fed into the foreground extraction network all require image preprocessing based on the foreground extraction network's input requirements. Image preprocessing methods include, but are not limited to, image scaling, image padding, image storage format conversion, and normalization.

[0146] In this embodiment, the foreground extraction network requires simultaneous input of a background image and an image to be processed, captured from the video. Therefore, the network of the present invention has two inputs, each with identical requirements, such as an image resolution of 512*288 and a channel count of 3. Therefore, the captured video image is first converted to a first intermediate image with a resolution of 512*288 through scaling and padding. The first intermediate image is then converted to RGB format to produce a second intermediate image. To speed up image processing, the R, G, and B values ​​of each pixel in the second intermediate image are normalized, for example, from 0-255 to 0-1, to produce an image that meets the input requirements of the foreground extraction network.

[0147] It should be noted that the present invention does not limit the training method of the convolutional neural network. As an example, the CDNet 2014 dataset or a self-built dataset can be used to train the convolutional neural network.

[0148] It should be noted that the reason for setting the frame skipping rule is to achieve a faster video processing speed while ensuring the video processing effect. Although several examples of frame skipping rules are listed here, those skilled in the art will understand that these examples should not constitute any limitation on the scope of protection of the present invention. Without changing the basic principles of the present invention, those skilled in the art can set the first frame skipping rule, the second frame skipping rule, the third frame skipping rule, and the fourth frame skipping rule according to the actual situation of the moving target in the video.

[0149] It should be noted that the target extraction method of the present invention includes but is not limited to the application scenario of video condensation, and is also applicable to other application scenarios requiring the extraction of moving targets, such as video summarization, moving target tracking, moving target recognition, etc.

[0150] Furthermore, the present invention also provides a target extraction device. Figure 5 As shown, the target extraction device 5 of the embodiment of the present invention mainly includes: a background acquisition module 51, an image acquisition module 52, a foreground target segmentation module 53 and a moving target extraction module 54.

[0151] As an example, the background acquisition module 51 is configured to perform the operation in step S101. The image acquisition module 52 is configured to perform the operations in steps S102, S104, and S105. The foreground object segmentation module 53 is configured to perform the operations in steps S103 and S106. The moving object extraction module 54 is configured to perform the operations in steps S107, S1071 to S1074, and S401 to S405.

[0152] Furthermore, the present invention also provides a computer device comprising a processor and a storage device. The storage device can be configured to store and execute a program for the moving target extraction method of the above-described method embodiment, and the processor can be configured to execute a program in the storage device, including but not limited to a program for executing the moving target extraction method of the above-described method embodiment. For ease of explanation, only the portions related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The moving target extraction device can be a control device device formed by various electronic devices.

[0153] Furthermore, the present invention also provides a storage medium that can be configured to store a program for executing the moving target extraction method of the above-mentioned method embodiment. The program can be loaded and executed by a processor to implement the above-mentioned moving target extraction method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present invention. The storage medium can be a storage device formed by various electronic devices. Optionally, the storage medium in the embodiment of the present invention is a non-transitory readable and writable storage medium.

[0154] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0155] It should be noted that, in the description of this application, the term "A and / or B" represents all possible combinations of A and B, such as only A, only B, or A and B.

[0156] It should be noted that the terms "first," "second," "third," "fourth," and the like in the specification and claims of the present invention and the accompanying drawings are used solely to distinguish similar objects and are not intended to describe or indicate a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present invention described herein can be practiced in an order other than that illustrated or described herein.

[0157] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A moving target extraction method, characterized in that: The method comprises: Get the background image; Acquire a first image from a first video to be processed according to a first frame skipping rule; Obtaining a first mask image sequence based on the first image and the background image; Acquire a second video to be processed from the first video to be processed according to the first mask image sequence; Acquire a second image from the second to-be-processed video according to a second frame skipping rule; obtaining a second mask image sequence based on the second image and the background image; Based on the second mask image sequence, a motion trajectory of the moving target is acquired.

2. The moving target extraction method according to claim 1, characterized in that: The step of "obtaining a first mask image sequence based on the first image and the background image" specifically includes sequentially inputting the first image and the background image into a trained foreground extraction network for semantic segmentation to obtain the first mask image sequence, wherein the first mask image sequence includes a plurality of first mask images; The step of "obtaining a second mask image sequence based on the second image and the background image" specifically includes sequentially inputting the second image and the background image into the trained foreground extraction network for semantic segmentation to obtain the second mask image sequence, where the second mask image sequence includes a plurality of second mask images; The foreground extraction network is a convolutional neural network; The first mask image and the second mask image have the same semantic information, and the semantic information includes that a position where the pixel value is 1 indicates that a foreground object exists, and a position where the pixel value is 0 indicates that the foreground object does not exist.

3. The moving target extraction method according to claim 2, characterized in that: The method further comprises: Marking the connected domains with pixel values ​​of 1 in the second mask image; According to the connected domain, position information of the foreground object in the second mask image is obtained, where the position information includes a foreground object ID and a rectangular box corresponding to the foreground object ID. The number of the foreground object IDs is one or more.

4. The moving target extraction method according to claim 3, characterized in that: The foreground target includes a first foreground target and a second foreground target, the first foreground target is the foreground target in the first result image, the second foreground target is the foreground target in the second result image, the first result image and the second result image are two adjacent second mask images in the second mask image sequence, and the second result image is the second mask image of the previous frame of the first result image, the position information includes first position information, second position information, and predicted position information, and the first position information, the second position information, and the predicted position information each include the respective foreground target IDs and rectangular boxes corresponding to the respective foreground target IDs; The step of “obtaining the motion trajectory of the moving target based on the second mask image sequence” specifically includes: Acquiring the first position information of the first foreground object; Obtaining the predicted position information of the second foreground object at the moment corresponding to the first result image; Obtaining an IoU value between the rectangular box in the first position information and the rectangular box in the predicted position information; The motion trajectory of the moving target is obtained according to the IoU value.

5. The moving target extraction method according to claim 4, characterized in that: The step of “obtaining the motion trajectory of the moving target according to the IoU value” specifically includes: Acquiring the second position information of the second foreground object; Using the IoU value as a weight of a KM algorithm, and obtaining a moving object matching degree between the first foreground object and the second foreground object through the KM algorithm; When the moving object matching degree is greater than or equal to a moving object matching degree threshold, it is determined that the first foreground object and the second foreground object are the same moving object; The motion trajectory of the moving target is acquired according to the first position information and the second position information.

6. The moving target extraction method according to claim 4, characterized in that: The foreground object further includes a third foreground object, the third foreground object is the foreground object in the third result image, the third result image is the second mask image of the first M frames of the second result image in the second mask image sequence, where M is an integer greater than or equal to 1; The step of “obtaining the predicted position information of the second foreground object at the moment corresponding to the first result image” specifically includes: Obtaining the current speed of the second foreground object; Acquiring a historical speed of the third foreground object, wherein the third foreground object and the second foreground object correspond to the same moving object; Obtaining a predicted speed of the second foreground object based on the current speed and the historical speed; Obtaining a center point of a rectangular frame of the predicted position information according to the predicted speed and a time difference between the second result image and the first result image; The predicted position information is obtained according to the center point of the rectangular frame of the predicted position information.

7. The moving target extraction method according to claim 6, characterized in that: The step of “obtaining the predicted position information according to the center point of the rectangular frame of the predicted position information” specifically includes: The width and height of the rectangular box of the predicted position information are respectively the average values ​​of the width and height of the rectangular boxes in the position information of each foreground target corresponding to the same moving target in the second result image and the second mask image N frames before the second result image, where N is an integer greater than or equal to 1.

8. The moving target extraction method according to claim 1, characterized in that: The methods for "get background image" include: Obtaining the initial background image; The step of "obtaining the initial background image" specifically includes: acquiring a third image from a time range specified within the initial stage of the first video to be processed according to a third frame skipping rule; Based on the third image, the initial background image is obtained by a median filtering algorithm.

9. The moving target extraction method according to claim 1, characterized in that: The methods for "get background image" include: maintaining the background image; The step of "maintaining the background image" specifically includes: Acquire a fourth image from the first video to be processed according to a fourth frame skipping rule; Sending the fourth image and the historical background image to a trained foreground extraction network to obtain a third mask image, wherein the historical background image is the background image before the time corresponding to the fourth image; The background image is updated according to the fourth image, the third mask image and the historical background image.

10. The moving target extraction method according to claim 9, characterized in that: During the process of acquiring the first mask image sequence, maintaining the background image; And / or, in the process of acquiring the second mask image sequence, the background image is maintained.

11. The moving target extraction method according to claim 2, characterized in that: The step of “obtaining a second video to be processed from the first video to be processed according to the first mask image sequence” specifically includes: Obtaining a foreground object ratio value of the first mask image according to the number of pixels whose pixels are 1 in the first mask image; When the foreground object ratio value is greater than or equal to the foreground object ratio threshold, it is determined that the foreground object exists in the first mask image; checking one by one whether the first mask images in the first mask image sequence contain the foreground object; When the foreground object exists in two adjacent first mask images, the first video to be processed within a time range corresponding to the two adjacent first mask images is acquired to obtain the second video to be processed.

12. A moving target extraction device, characterized in that: The device comprises: A background acquisition module is configured to perform the following operations: Get the initial background image, maintaining the background image; An image acquisition module, wherein the image acquisition module is configured to perform the following operations: According to the first frame skipping rule, a first image is obtained from the first video to be processed. According to the first mask image sequence, a second video to be processed is obtained from the first video to be processed, Acquire a second image from the second to-be-processed video according to a second frame skipping rule; A foreground object segmentation module is configured to perform the following operations: Based on the first image and the background image, a first mask image sequence is obtained, obtaining a second mask image sequence based on the second image and the background image; A moving target extraction module is configured to obtain a motion trajectory of the moving target according to the second mask image sequence.

13. The moving target extraction device according to claim 12, characterized in that: The foreground object segmentation module is configured to perform the following specific operations: Submitting the first image and the background image to a trained foreground extraction network for semantic segmentation in sequence to obtain a first mask image sequence, where the first mask image sequence includes multiple first mask images; sequentially sending the second image and the background image to the trained foreground extraction network for semantic segmentation to obtain a second mask image sequence, where the second mask image sequence includes multiple second mask images; The foreground extraction network is a convolutional neural network; The first mask image and the second mask image have the same semantic information, and the semantic information includes that a position where the pixel value is 1 indicates that a foreground object exists, and a position where the pixel value is 0 indicates that the foreground object does not exist.

14. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, wherein: The program code is suitable for being loaded and run by the processor to execute the moving target extraction method according to any one of claims 1 to 11.

15. A storage medium, adapted to store a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the moving target extraction method according to any one of claims 1 to 11.

Images