An image semantic label making method combining ViBe and RGBD image information

Abstract

The application relates to a kind of image semantic label making methods of combining ViBe with RGBD image information, comprising:1) video acquisition and pretreatment;2) spatial registration of image sequence, obtain corresponding transformation matrix;3) motion target detection and binarization are carried out to gray image sequence;4) extract the key frame and Id of binarized image;5) extract the edge of depth image key frame, according to depth information, combine maximum connected filter algorithm and edge region traversal algorithm, on the foreground area in foreground segmentation image, edge correction is carried out and shadow noise is removed;6) label classification and pixel filling are carried out on the foreground area thereof;7) the obtained label is saved to the corresponding folder.The method of the application extracts key frame, removes image shadow using depth information, extracts target key frame foreground and fills pixel combining sliding window idea, and gives label, without manual image segmentation, reduces the complexity of semantic label making, improves the precision and efficiency of label making.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a method for making image semantic labels combined with ViBe and RGBD image information. BACKGROUND

[0002] The problem solved by the background difference method involves the comparison of a frame of a video sequence with a reference background frame or model. This comparison, which is called foreground detection, is a classification process that divides the image into two complementary pixel sets. The ViBe algorithm has stronger robustness than the inter-frame difference method and the Gaussian mixture model (GMM), and its use to detect the difference between foreground and background is an effective way to detect dynamic targets in video sequences in computer vision. The traditional ViBe algorithm converts color images into grayscale images to improve the rate of foreground segmentation. This method has strict requirements for the lighting conditions of the environment. If there is a shadow in the environment, the traditional ViBe algorithm will also regard the shadow as a dynamic target, thereby reducing the recall rate and accuracy of dynamic target detection. Convolutional neural networks (CNN) are a highly efficient recognition method that has developed rapidly since its inception and has attracted widespread attention. In recent years, in-depth research has made it widely used, especially in the field of pattern recognition. For convolutional neural networks, the training data set of the model is almost manually segmented images with corresponding labels, which requires a lot of manpower. Although open source data sets are gradually diversifying, it is still difficult to directly obtain the data of the training target required by oneself. SUMMARY

[0003] The purpose of the present application is to address the high complexity and low efficiency of existing semantic label making, and to provide a method for making image semantic labels combined with ViBe and RGBD image information, which reduces the complexity of semantic label making and improves the accuracy and efficiency of label making. The main features of the method are the extraction of key frames, the use of depth information to remove image shadows, the extraction of target key frame foreground combined with the sliding window idea and the filling of pixels, and the given label without manual image segmentation.

[0004] To achieve the above purpose, the present application provides the following technical scheme:

[0005] A method for making image semantic labels combined with ViBe and RGBD image information, comprising the following steps:

[0006] Step S1, video acquisition and preprocessing to obtain an RGB image sequence and a depth image sequence;

[0007] Step S2, spatial registration of the RGB image sequence and the depth image sequence obtained in step S1 to obtain an aligned image sequence and a transformation matrix corresponding to the two sequence image groups.

[0008] Step S3: After converting the RGB image obtained in step S1 to grayscale, the ViBe algorithm is used to detect moving targets in the grayscale image sequence to obtain the foreground segmentation image of the image, and the image is binarized.

[0009] Step S4: Use the sliding window concept to extract keyframes from the moving targets in the video image to obtain the keyframe image ID, and then extract the corresponding grayscale image keyframes, RGB image keyframes, and depth image keyframes in sequence.

[0010] Step S5: Binarize the grayscale image keyframes and depth image keyframes obtained in step S4 to distinguish the foreground and background. At the same time, extract the edges of the depth image keyframes. Using the depth information in the depth image, combine the maximum connected filtering algorithm and the edge region traversal algorithm to locate the foreground region in the foreground segmentation image obtained in step S3 for edge correction and removal of shadow noise.

[0011] Step S6: Perform label classification on the binarized image from step S5 and fill the pixels in the region represented by the foreground.

[0012] Step S7: Place the tags obtained in step S6 into the corresponding folders through the program terminal and human judgment.

[0013] Further, the video acquisition and preprocessing described in step 1 yields an RGB image sequence and a depth image sequence, as detailed below:

[0014] Step S11: Use an RGBD camera to capture video, place the items in the pre-arranged background area, and remove the items from the background area after a short time. This process includes placing all the items that need to be labeled in different poses until all items have completed the above process, then stop capturing and obtain a video file.

[0015] Step S12: Create a video stream control object for the file, read the current frame specified by the video stream in a loop, convert the frame into BGR mode and depth image mode to obtain RGB image sequence and depth image sequence, and save them in the specified folder in order until the video image frame ends, and close the video stream object.

[0016] Further, in step 2, the RGB image sequence and the depth image sequence are spatially registered to obtain aligned image sequences and transformation matrices corresponding to the two image sequence groups, specifically as follows:

[0017] Spatial alignment is performed on the depth image and the corresponding color image acquired by the RGBD camera. This involves selecting four or more matching point pairs in both the depth image and the corresponding color image, representing (X, Y, Z) points in the depth image.D ,Y D )={(x D1 ,y D1 ),

[0018] (x D2 ,y D2 ), ..., (x Dn ,y Dn )}, corresponding to (X) in the color image C ,Y C )={(x C1 ,y C1 ), (x C2 ,y C2 ), ...,

[0019] (x Cn ,y Cn )}, where n≥4; the homography matrix is ​​solved by the Direct Linear Transform (DLT) method.

[0020] Further, in step 3, after converting the RGB image to grayscale, the ViBe algorithm is used to detect moving targets in the grayscale image sequence to obtain a foreground segmentation image, which is then binarized. This process specifically includes the following steps:

[0021] Step S31: Initialize and construct the sample dataset. Based on the experience of the ViBe algorithm, in the first frame of the image sequence, a random strategy is used to select 20 pixels from the 8-neighborhood N0(x,y) of the center pixel to build the background model.

[0022] Step S32: The image sequence is processed by ViBe to process each frame after the background frame. k (x,y) is used for foreground pixel detection, and the Euclidean distance between the pixel and the corresponding background sample set is calculated. k (x,y);

[0023] Step S33: Count the number of foreground pixels in the image frames where dynamic objects are determined to exist, and record the number of foreground pixels in the k-th frame as...

[0024] Further, in step 4, keyframe extraction is performed on the moving target in the video image to obtain the keyframe image ID. The corresponding grayscale image keyframes, RGB image keyframes, and depth image keyframes are then extracted sequentially. The specific process is as follows:

[0025] Step S41: Traverse each frame after the background sample set, and set the sliding window dataset. L is the length of the sliding window, i.e. the window can store L frames of images; for the sliding window, if the sliding window is not filled, the next frame of image is traversed until the sliding window is filled, i.e. When the sliding window is filled, the average value of the dynamic pixels of each frame in the sliding window is calculated;

[0026] Step S42, the frame after the sliding window is recorded as the k α frame, and the absolute value of the difference between the k α frame and the average value of the dynamic pixels of each frame in the sliding window is calculated as the basis for extracting the key frame;

[0027] Step S43, if the sliding window has been filled and the difference between the number of foreground pixel of each frame and the average number of foreground pixels in the sliding window is less than or equal to the stability threshold n st , it is determined that the sliding window is a stable sliding window, further, if the average number of foreground pixels in the sliding window is greater than or equal to the target pixel threshold n TG , it is determined that the sliding window is a stable dynamic target window, otherwise it is determined to be a stable background window;

[0028] Step S44, when the window is given the attribute of dynamic or background, the number of foreground pixels of the next frame is counted first, if the difference between the value and the average number of foreground pixels in the window is still less than or equal to the stability threshold n st , it is considered that the frame is also of the same nature as the image frames in the sliding window, at this time the length of the sliding window is increased by one, and the frame is placed at the end of the sliding window; if the difference between the value and the average number of foreground pixels in the window is greater than the stable pixel threshold n TG , the frame is discarded, the sliding window stops sliding, if the sliding window is determined to be a dynamic target window, the middle frame in the sliding window is extracted as a key frame, if it is a background window, the middle frame is extracted as the basis for background updating;

[0029] Step S45: record the key frame Id obtained in step S44, and sequentially extract the depth image I kD and the grayscale image I kG with the same frame Id.

[0030] Further, the process of removing shadow noise while correcting the edge in step 5 is:

[0031] Step S51, the dynamic pixel position obtained by div k (x, y) < R in step S32 is marked as foreground, and binarization is performed in another image space I' kC , i.e. the corresponding pixel points in the binary image are filled with 1, and the remaining pixel points are filled with 0;

[0032] Step S52, for the depth image, only binarize the depth image key frame, combine the frame difference method to obtain the target frame difference image which is different from the depth image background frame, and combine the I' kC (x,y) to obtain the depth key frame binary image I' kD (x,y) and the filtered I' kC (x,y);

[0033] Step S53, according to the homography matrix obtained in step S2, map the depth image into the color image or the grayscale image to obtain the depth image which is aligned with the color image;

[0034] Step S54, perform edge extraction on I' kD2C (x,y);

[0035] Step S55, use a detection area kernel N with a size of n*n in the I' kD2C (x,y) edge to traverse the I' kD2C (x,y) edge, and meanwhile, correct the foreground pixel area in the I' kC (x,y) which is in the same position as the I' kD2C (x,y) edge;

[0036] Step S56, fill the closed area in the edge pixel detection result obtained in step S55 with 1 to obtain the key frame binary image which is aligned with the color image and has the shadow removed.

[0037] Further, the label classification and the pixel filling of the binary image in step 6 include: pre-establishing a label folder and an RGB folder related to the article, and filling the non-0 area of the binary image obtained in step 5 with the corresponding label pixel according to the formula.

[0038] Compared with the prior art, the method has the advantages that:

[0039] The method extracts the key frame, removes the image shadow by using the depth information, extracts the target key frame foreground and fills the pixel by combining the sliding window idea, and gives the label, so that the image does not need to be manually segmented, the complexity of the semantic label production is reduced, and the precision and efficiency of the label production are improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a flowchart of the image semantic label production method combining the ViBe and the RGBD image information according to the present application;

[0041] Figure 2 is a scene diagram of the label production in the embodiment of the present application;

[0042] Figure 3 is part of the image sequence of the article placement process in the embodiment of the present application;

[0043] Figure 4 is a schematic diagram of color image and depth image registration in the embodiment of the present application;

[0044] Figure 5 is a schematic diagram of key frame depth image and grayscale image edge extraction, filtering and key frame edge correction in the embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0046] The present technology is directed to the professional terms that can be understood by the personnel in the corresponding technical field, and unless otherwise defined, all terms (including technical terms and scientific terms) in the present application have meanings similar to those understood by the ordinary skilled person in the art. It should also be understood that terms such as those defined in general dictionaries should be understood as having meanings consistent with the prior art, and unless specifically defined, they should not be interpreted as having idealized or overly formal meanings.

[0047] Embodiment: see Figures 1-5 .

[0048] As Figure 1 shown, a method for making image semantic labels combined with ViBe and RGBD image information, the specific implementation process includes the following steps:

[0049] Step S1, video acquisition and preprocessing, get RGB image sequence and depth image sequence;

[0050] Step S11, place the article into the background area arranged in step S1, as Figure 2 shown, remove the article from the background area after a short time, this process includes placing all articles that need to be labeled in different poses, as Figure 3 shown, until all articles have completed the above process, stop collecting, get the video file;

[0051] Step S12, create a video stream control object for the file, read the current frame specified by the video stream in a loop, convert the frame to BGR mode and depth image mode, get the RGB image sequence and depth image sequence, and save them in the specified folder in order until the end of the video image frame, close the video stream object;

[0052] Step S2, register the RGB image sequence and depth image sequence obtained in step S1 in space, get the aligned image sequence and the transformation matrix corresponding to the two sequence image groups, as Figure 4are shown, wherein (a) is a depth image, (b) is a gray image, (c) is a position map of the depth image after transformation on the gray image, and (d) is a superimposed schematic diagram of the depth image and the gray image;

[0053] Step S21, the depth image and the corresponding color image collected by the RGBD camera are spatially aligned, that is, four or more matching point pairs are selected in the depth image and the corresponding color image, respectively, as (X D ,Y D ) = {(x D1 ,y D1 ), (x D2 ,y D2 ), …, (x Dn ,y Dn )} in the depth image and (X C ,Y C ) = {(x C1 ,y C1 ), (x C2 ,y C2 ), …,

[0054] (x Cn ,y Cn )} in the corresponding color image, wherein n≥4; the following relationship exists between the matching point pairs:

[0055]

[0056] In the above formula, H is a homography matrix, H ij (i,j∈1,2,3) is a parameter of the homography matrix, and then:

[0057]

[0058] Let H 33 = 1, and the non-zero factor is removed and arranged to obtain:

[0059]

[0060] The homography matrix is solved by a direct linear transformation (DLT) method, and the following is obtained:

[0061]

[0062] Step S3, the RGB image obtained in step S1 is subjected to gray scale conversion, and then a ViBe algorithm is used to detect a moving target in the gray image sequence to obtain a foreground segmentation image of the image, and the image is binarized;

[0063] Step S31, initialize the construction of sample data set, that is, according to the experience of ViBe algorithm, set 20 pixel points in the 8-neighborhood N0(x, y) of the center pixel point in the first frame of the image sequence to establish the background model by using the random strategy:

[0064] M0(x, y) = {V0((x, y) | (x, y) ∈ N0(x, y)} (5)

[0065] Wherein M0(x, y) is the pixel point in the initial sample set, N0(x, y) is the direct neighborhood point in the pixel color space, and V0(x, y) is the pixel value of the (x, y) pixel point in the initial sample;

[0066] Step S32, perform foreground pixel detection on each frame I k (x, y) of the image sequence after the background frame by ViBe, calculate the Euclidean distance div k (x, y) between the pixel point and the background sample set corresponding to its position:

[0067] div k (x, y) = |I k (x, y) - M0(x, y)|

[0068] Step S33, count the foreground pixel points of the image frame determined to exist dynamic object, and record the number of foreground pixel points in the kth frame as

[0069] Step S4, use the Sliding Window (sliding window) idea to extract the key frame of the video image motion target, obtain the key frame image Id, and extract the corresponding gray image key frame, RGB image key frame and depth image key frame;

[0070] Step S41, traverse each frame after the background sample set, set the sliding window data set L is the length of the sliding window, that is, the window can store L frames of images; then for the sliding window, if the sliding window is not filled, traverse the next frame of image until the sliding window is filled, that is, When the sliding window is filled, calculate the average value of each frame pixel in the sliding window:

[0071]

[0072] Step S42, record the next frame of the sliding window as the kth α frame, and has the following formula:

[0073]

[0074] Calculate the kth αThe absolute value of the difference between each frame and the average value of the dynamic points in the sliding window is taken as the basis for extracting the key frame.

[0075] Step S43, if the sliding window has been filled and the difference between the number of foreground point pixels of each frame and the average number of foreground point pixels in the sliding window is less than or equal to the stability threshold n st , it is determined that the sliding window is a stable sliding window, and further, if the average number of foreground point pixels in the sliding window is greater than or equal to the target pixel threshold n TG , it is determined that the sliding window is a stable dynamic target window, otherwise it is determined to be a stable background window.

[0076] Step S44, after the window is given the attribute of dynamic or background, the number of foreground point pixels of the next frame is first counted, and if the difference between the value and the average number of foreground point pixels in the window is still less than or equal to the stability threshold n st , it is considered that the frame is also of the same type as the image frames in the sliding window, at this time the length of the sliding window is increased by one, and the frame is placed at the end of the sliding window; if the difference between the value and the average number of foreground point pixels in the window is greater than the stable pixel threshold n TG , the frame is discarded, the sliding window stops sliding, and if the sliding window is determined to be a dynamic target window, the middle frame in the sliding window is extracted as a key frame, and if it is a background window, the middle frame is extracted as a basis for background updating, i.e.

[0077]

[0078] Step S45, the key frame Id obtained in step S44 is recorded, and the depth image I kD and the gray image I kG with the same frame Id are extracted;

[0079] Step S5, the gray image key frame and the depth image key frame obtained in step S4 are binarized to distinguish foreground and background, as shown in (d) of Figure 5 , and the edge of the depth image key frame is extracted, and the foreground region in the foreground segmentation image obtained in step three is located using the depth information in the depth image, combined with the maximum connected filtering algorithm and the edge region traversal algorithm, for edge correction and shadow noise removal;

[0080] Step S51, the dynamic point pixel position obtained by div k (x,y) in step S32 is marked as foreground, and binarization is performed in another image space I' kC , i.e. the corresponding pixel points in the binary image are filled with 1, and the rest of the pixel points are filled with 0, as shown in formula (3), to obtain:

[0081]

[0082] In the above formula, Num(div k (x,y)<R) represents the number of pixels satisfying the condition div k (x,y)<R;

[0083] Step S52, for the depth image, only the depth image key frame is binarized, and the frame difference method is combined to obtain the target frame difference image which is different from the depth image background frame, as shown in (a) of FIG. 6. Figure 5 (x,y) is subjected to maximum connected filtering and median filtering to obtain the depth key frame binary image I′ kC (x,y) and the filtered I′ kD (x,y), as shown in (b) and (e) of FIG. 6, respectively. kC (x,y), as shown in (b) and (e) of FIG. 6, respectively. Figure 5

[0084] Step S53, according to the homography matrix obtained in step S2, the depth image is mapped into the color image or the gray image to obtain the depth image which is aligned with the color image, that is:

[0085] I′ kD2C (x,y) = HI′ kD (x,y) (11)

[0086] Step S54, edge extraction is performed on I′ kD2C (x,y) to obtain the result as shown in (c) of FIG. 6. Figure 5

[0087] Step S55, in the edge of I′ kD2C (x,y), a detection area kernel N with a size of n×n is used to traverse the edge of I′ kD2C (x,y), and at the same time, the foreground pixel area in I′ kC (x,y) which is at the same position as the edge of I′ kD2C (x,y) is corrected, that is:

[0088]

[0089] In the above formula, I′ kC (N(i,j)) represents the area i,j∈{1,2,…,n} in I′ kC which is at the same position as the detection area kernel N of I′ kD2C (x,y), and the process is shown in (f)-(h) of FIG. 6. Figure 5

[0090] ​​​Step S56, fill the closed region in the edge pixel detection result obtained in step S55 with 1, to obtain a key frame binary image aligned with the color image and de-shadowed, the result of which is as follows Figure 5 as shown in (i);

[0091] Step S6, label classification is performed on the binary image in step S5, and the region represented by the foreground is filled with pixels;

[0092] Step S61, pre-establish a label folder and an RGB folder related to the article, and fill the non-0 region of the binary image obtained in step S5 with corresponding label pixels according to the formula:

[0093]

[0094] In the above formula, label[R, G, B] is the pixel value of each channel R, G and B in the label corresponding to the key frame. Through this step, the label image I kL (x, y) is obtained, as shown in (l); ​

[0095] Step S7, put the label image obtained in step S6 into the corresponding folder through program terminal and human judgment.

[0096] The preferred implementation method of the present application is described in detail above, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.​

Claims

1. An image semantic label making method combining ViBe with RGBD image information, characterized in that, Comprise the following steps: Step S1, video acquisition and pretreatment, get RGB image sequence and depth image sequence; Step S2, the RGB image sequence and depth image sequence obtained in step S1 are registered in space, and the aligned image sequence and the corresponding transformation matrix of the two sequence image groups are obtained; Step S3, after the RGB image obtained in step S1 is converted to grayscale, the ViBe algorithm is used for motion target detection on the grayscale image sequence, the foreground segmentation image of the image is obtained, and the image is binarized; Step S4, using the sliding window Sliding Window idea to extract the key frame of the video image motion target, get the key frame image Id, in turn extract the corresponding grayscale image key frame, RGB image key frame, depth image key frame; Step S5, the grayscale image key frame and depth image key frame obtained in step S4 are binarized to distinguish foreground and background, and the edge of the depth image key frame is extracted, the depth information in the depth image is used, combined with the maximum connected filter algorithm and the edge region traversal algorithm to locate the foreground region in the foreground segmentation image obtained in step S3, for edge correction and shadow noise removal; Step S6, the image binarized in step S5 is classified by label and the region represented by the foreground is filled with pixels; Step S7, the label obtained in step S6 is put into the corresponding folder through the program terminal and human judgment.

2. The image semantic label making method combining ViBe with RGBD image information according to claim 1, characterized in that, In step 1, the video is collected and pretreated to obtain the RGB image sequence and the depth image sequence, and the specific steps are as follows: Step S11, an RGBD camera is used to collect video, the object is placed in the pre-arranged background area, and after a short time, the object is removed from the background area, this process includes placing all objects that need to be labeled in different poses, until all objects complete the above process, stop collecting, and get the video file; Step S12, a video stream control object is established for the file, the current frame of the video stream is read in a loop, the frame is converted to BGR mode and depth image mode, the RGB image sequence and the depth image sequence are obtained, and saved in the specified folder in order until the end of the video image frame, and the video stream object is closed.

3. The image semantic label making method combining ViBe with RGBD image information according to claim 1, characterized in that, In step 2, the RGB image sequence and the depth image sequence are registered in space, and the aligned image sequence and the corresponding transformation matrix of the two sequence image groups are obtained, which is specifically: Step S21, the depth image and the corresponding color image collected by the RGBD camera are spatially aligned, that is, four or more matching point pairs are selected in the depth image and the corresponding color image, respectively, (X D ,Y D ) = {(x D1 , y D1 ), (x D2 , y D2 ),..., (x Dn , y Dn )} in the depth image, (X C ,Y C ) = {(x C1 , y C1 ), (x C2 , y C2 ),..., (x Cn , y Cn )} in the corresponding color image, where n≥4; the matching point pairs satisfy the following relationship: In the above equation, H is a homography matrix, H ij (i, j e 1, 2, 3) are parameters of the homography matrix, then we have: Let H 33 = 1, removing its non-zero factor, we have: The homography matrix is solved by direct linear transformation (Direct Linear Transform, DLT), and the following is obtained:

4. The image semantic label making method combining ViBe with RGBD image information according to claim 1, characterized in that, In step 3, after the RGB image is converted to grayscale, the ViBe algorithm is used for motion target detection on the grayscale image sequence, the foreground segmentation image of the image is obtained, and the image is binarized, which includes the following steps: Step S31, initialize the construction of sample data set, that is, according to the experience of ViBe algorithm, 20 pixel points in the 8-neighborhood N0(x, y) of the center pixel point are selected from the first frame of the image sequence to establish the background model: M0(x, y)={V0((x, y)|(x, y)∈N0(x, y)} (5) Wherein M0(x, y) is a pixel point in the initial sample set, N0(x, y) is a directly adjacent point in the pixel color space, and V0(x, y) is a pixel value of the pixel point (x, y) in the initial sample; Step S32, each frame I of the image sequence after the background frame is detected by ViBe k (x, y) for foreground pixel detection, the Euclidean distance div between the pixel point and the background sample set corresponding to its position is calculated k (x, y): div k (x, y) = |I k (x, y) - M0(x, y) | (6) Step S33, foreground pixel point statistics is performed on the image frame in which it is determined that there is a dynamic object, and the number of foreground pixels in the kth frame is recorded as Nk.

5. The image semantic label making method combining ViBe with RGBD image information according to claim 1, characterized in that, The key frame extraction of the moving target of the video image in step 4 obtains a key frame image Id, and corresponding gray image key frames, RGB image key frames and depth image key frames are sequentially extracted, and the specific process is as follows: Step S41, for each frame after traversing the background sample set, set the sliding window data set L is the length of the sliding window, that is, the window can store L frames of images; then for the sliding window, if the sliding window is not filled, the next frame of image is traversed until the sliding window is filled, that is, When the sliding window is filled, the average value of each frame of image in the sliding window is calculated: Step S42, record the frame after the sliding window as the kth frame α frame, according to the following formula: The kth frame is calculated α The absolute value of the difference between the frame and the average value of the dynamic pixels in each frame in the sliding window is taken as the basis for extracting the key frame. Step S43, if the sliding window has been filled, and the difference between the foreground pixel number of each frame and the average foreground pixel number in the sliding window is less than or equal to the stability threshold n st , then it is determined that the sliding window is a stable sliding window, further, if the average foreground pixel number in the sliding window is greater than or equal to the target pixel threshold n TG , then it is determined that the sliding window is a stable dynamic target window, otherwise it is determined as a stable background window; Step S44, when the window is given dynamic or background attribute, the foreground pixel number of the next frame is counted first, if the difference between the value and the average value of the foreground pixel number in the window is still less than or equal to the stable threshold n st , it can be considered that the frame is also a frame of the same nature as the image frames in the sliding window, at this time the length of the sliding window increases by one, and the frame is placed at the end of the sliding window; if the difference between the value and the average value of the foreground pixel number in the window is greater than the stable pixel threshold n TG for one frame, the frame is discarded, the sliding window stops sliding, if the sliding window is determined as a dynamic target window, the middle frame in the sliding window is extracted as a key frame, if it is a background window, the middle frame is extracted as the basis for background updating, that is: Step S45, records the key frame Id obtained in step S44, and extracts the depth image I of the same frame Id accordingly kD and the gray scale image I kG .

6. The image semantic label making method combining ViBe with RGBD image information according to claim 1, characterized in that, The process of the edge correction and the removal of the shadow noise in step 5 is as follows: Step S51, according to div in step S32 k (x, y) < R, the dynamic point pixel position obtained, is marked as foreground, and another image space I' is obtained kC is binarized, that is, the pixel point at the corresponding position in the binary image is filled with 1, and the rest of the pixel points are filled with 0; Step S52, for the depth image, only binarize the depth image key frame, combine the frame difference method to obtain the target frame difference image which is different from the depth image background frame, and combine the I′ kC (x, y) to obtain the depth key frame binary image I′ kD (x, y) and the filtered I′ kC (x, y); Step S53, according to the homography matrix obtained in step S2, the depth image is mapped into the color image or the gray image, and a depth image aligned with the color image is obtained, that is: I′ kD2C (x, y) = HI′ kD (x, y) (11) Step S54, edge extraction is performed on I' kD2C (x, y) Step S55, in I' kD2C (x,y) edges using a detection region kernel N of size n x n, on I' kD2C (x,y) edges, while modifying the foreground pixel region in I' kC (x,y) edges, while modifying the foreground pixel region in I' kD2C (x,y) edges, i.e.: In the above formula, I' (x, y) represents the intensity of the pixel at the position (x, y) in the image I'. kC (N(i, j)) represents the intensity of the pixel at the position (i, j) in the image N. kC In the above formula, I' (x, y) represents the intensity of the pixel at the position (x, y) in the image I'. kD2C (x, y) detection region nucleus N region i, j ∈ {1, 2,..., n}; Step S56, the closed region in the edge pixel detection result obtained in step S55 is filled with 1, and a key frame binary image aligned with the color image and free of shadow is obtained.

7. The image semantic label making method combining ViBe with RGBD image information according to claim 1, characterized in that, The label classification of the binary image in step 6 and the pixel filling of the region represented by the foreground thereof include: A label folder and an RGB folder related to the article are pre-established, and the non-0 region of the binary image obtained in step S5 is filled with the corresponding label pixel according to the following formula: The process of the edge correction and the removal of the shadow noise in step 5 is as follows: Step S53, according to the homography matrix obtained in step S2, the depth image is mapped into the color image or the gray image, and a depth image aligned with the color image is obtained, that is: Step S56, the closed region in the edge pixel detection result obtained in step S55 is filled with 1, and a key frame binary image aligned with the color image and free of shadow is obtained. The label classification of the binary image in step 6 and the pixel filling of the region represented by the foreground thereof include: A label folder and an RGB folder related to the article are pre-established, and the non-0 region of the binary image obtained in step S5 is filled with the corresponding label pixel according to the following formula: In the above formula, label[R, G, B] is the pixel value of each channel R, G, B in the label corresponding to the key frame, and the label image I obtained through the key frame detection is obtained through this step kL (x, y).

Images