A method and system for constructing a scene-adaptive video dataset

Abstract

The application provides a scene adaptive video dataset construction method and system, based on various behavior video data of existing scenes, semantic analysis and similarity matching are performed on the target scene and the existing scene, a pre-trained behavior classification model is used to classify the existing behavior data, target behavior video data is obtained, a scene adaptive mechanism is proposed, a target tracking algorithm based on video sequence content is designed, behavior target sequence information is obtained, a background subtraction algorithm is used to extract the foreground target from the target sequence detection frame, and then scene adaptive conversion, target sequence migration and optimization are performed according to the attributes and content of the target scene, high-quality video data of the required target behavior under the target scene is generated. The application can quickly realize adaptive data generation of various scenes, does not require a large amount of video data provided by the target scene, has a lower requirement for the data amount of the target scene, and is helpful for behavior detection and recognition and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and relates to a scene-adaptive video dataset construction method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In recent years, with the research on deep learning theory and the continuous improvement of computer computing power, computer vision technology has developed rapidly. Behavior detection and recognition, as an important branch of computer vision, is also booming and widely used in various fields of life such as intelligent supervision, unmanned driving, social security, etc., providing more convenient and efficient life for the public, and greatly promoting the process of combining with other computer technologies.

[0004] In order to realize the automatic and accurate recognition of behaviors in the monitoring video by the computer, a large number of video datasets with behavior labels need to be obtained first. For a behavior recognition network model, the more training data it has, the more important it is to the training of the model and the improvement of the accuracy of the model. Since in real life, the application scenarios of some specific behaviors (such as fighting, gathering, and sitting for a long time) are various, however, at present, the existing behavior video dataset only contains a limited number of scenes. In order to realize the detection and recognition of some behaviors in various scenes, behavior video data in a specific scene need to be obtained for model training, and there is less behavior data in the existing scene (or the same as the existing scene) that meets the training requirements. In order to make the training effect close to the effect of the same scene, a large amount of behavior data in similar scenes is needed for training.

[0005] For monitoring videos, the moving objects or targets in the video are generally referred to as foreground, and the remaining part after removing the foreground is considered as background. At present, foreground extraction and target migration for pictures are relatively common, but the research on target migration for video sequences is relatively less. Since behavior recognition detection is realized based on video sequences, generating video sequences of various behaviors is of great significance to the expansion of behavior data quantity in a specific scene.

[0006] For a model for behavior detection and recognition, the more abundant the behavior dataset used for training is, the more beneficial it is to the improvement of the recognition accuracy of various behaviors. Therefore, in order to make the model recognition performance more outstanding and efficient, a large amount of training data is usually needed. Since the behaviors to be recognized in different scenes are different, it is very difficult to construct a large-scale, high-quality behavior dataset that can meet the needs of different scenes in the actual application process, and it is difficult to meet the efficiency and speed. SUMMARY

[0007] To solve the above problems, the present application provides a scene-adaptive video dataset construction method and system, which can quickly generate adaptive data for multiple scenes, without the need for the target scene to provide a large amount of video data, and has a lower requirement for the amount of target scene data, which is helpful for behavior detection and recognition applications.

[0008] According to some embodiments, the present application adopts the following technical solutions:

[0009] A scene-adaptive video dataset construction method includes the following steps:

[0010] Based on various behavior video data of existing scenes, the target scene and the existing scene are subjected to semantic analysis and similarity matching;

[0011] The behavior data of the existing scene with a similarity exceeding a predetermined value is obtained, a pre-trained behavior classification model is used for classification, and target behavior video data is obtained;

[0012] The target behavior video data is subjected to target tracking to obtain behavior target sequence information, and a target sequence detection frame is subjected to foreground target extraction;

[0013] According to the properties and content of the target scene, the behavior target sequence is subjected to scene-adaptive conversion to obtain a target sequence more similar to the target scene;

[0014] The target sequence and the target background image are synthesized and spliced to obtain a video dataset of the required behavior in the target scene with continuous behavior actions.

[0015] As an optional implementation, the specific process of performing semantic analysis and similarity matching on the target scene and the existing scene based on various behavior video data of the existing scene includes: performing semantic analysis on the target scene and the existing scene to obtain scene content attribute features, and performing scene similarity matching on the target scene and the existing scene to obtain similar existing scene behavior data of the target scene.

[0016] As an optional implementation, the specific process of using a pre-trained behavior classification model for classification includes: feeding the behavior video data of the similar existing scene into a behavior classification network model for batch classification, sequentially encoding the video stream, calculating the features through convolution, taking the features as input, generating different lengths of time ranges that may contain behaviors, filtering the time ranges, converting the features and different lengths of time range segments into fixed-length feature maps, and finally classifying the behavior categories and predicting the refined behavior boundaries.

[0017] As an optional implementation method, the specific process of target tracking for target behavior video data includes: using a target tracking algorithm to track the moving entities in the video, obtaining the detection frame and coordinate position of each moving target frame by frame or every other frame, and saving the target detection frame of each moving object separately to obtain a continuous behavior target sequence detection frame image with time-related information.

[0018] As an optional implementation method, the specific process of extracting foreground targets from target sequence detection frames includes using a background subtraction method to separate the foreground information and background information in each target sequence detection frame image in each frame, remove the image background, and obtain the temporally correlated behavioral target sequence image information of interest.

[0019] As a further limitation, the concept of transparency channel is introduced, and it is considered that the image consists of foreground, background and transparency. The transparency is the probability that the pixel belongs to the foreground. The auxiliary information generated by the processed original image and the adjacent frame is combined with the background Figure One The image is fed into the encoder, and then the transparency and foreground are output through the residual structure and decoder module, continuously reducing the error between the transparency, foreground and original image color. After several iterative trainings, the foreground target sequence of interest is finally extracted.

[0020] As an optional implementation method, the specific process of synthesizing and splicing the target sequence with the target background image includes synthesizing the target sequence with the target background image frame by frame to obtain a synthetic image, and splicing the behavior synthetic image frame by frame to obtain video data of specific behavior in the target scene with continuous behavior actions.

[0021] A system for constructing a scene-adaptive video dataset, comprising:

[0022] A scene semantic analysis and matching module is configured to perform semantic analysis and similarity matching between the target scene and the existing scene based on various behavior video data of the existing scene;

[0023] The existing scene behavior video classification module is configured to obtain behavior data of existing scenes whose similarity exceeds a predetermined value, classify them using a pre-trained behavior classification model, and obtain target behavior video data;

[0024] The target sequence information extraction module is configured to track the target behavior video data, obtain the behavior target sequence information, and extract the foreground target from the target sequence detection frame;

[0025] A scene adaptive conversion module is configured to perform scene adaptive conversion on the behavior target sequence according to the attributes and content of the target scene to obtain a target sequence that is more similar to the target scene;

[0026] The behavior video data generation module is configured to synthesize and splice the target sequence and the target background image to obtain video data sets of the required behavior in the target scene with continuous behavior actions.

[0027] A computer readable storage medium, wherein a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and to execute the steps in the method.

[0028] A terminal device, comprising a processor and a computer readable storage medium, the processor being configured to implement instructions, and the computer readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded by the processor and to execute the steps in the method.

[0029] Compared with the prior art, the present application has the following beneficial effects:

[0030] According to the present application, a video sequence containing a specified behavior in a single target scene image can be generated, and a large amount of video data provided by the target scene is not required, and the requirement for the amount of data of the target scene is low.

[0031] The present application migrates a foreground target sequence with continuous actions in a behavior video to a target scene image to form a video data with continuous behavior actions, and realizes migration of a large number of target sequences instead of single target migration, thereby breaking through the limitation of the single target object migration mode.

[0032] The present application proposes a scene similarity matching and scene adaptive mechanism, matches behavior video data in a similar scene for a target scene, analyzes image properties and content features in the target scene, and performs scene adaptive conversion on a target sequence to limit the activity area of a target object, so that the problems of low fusion degree of the target sequence and the target background and the problem of non-compliance with the actual motion trajectory can be avoided, and the scene is not limited, and adaptive data generation in multiple scenes can be realized.

[0033] The advantages of the additional aspects of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0034] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation on the present application.

[0035] Figure 1 The present application is a schematic diagram of an operation process;

[0036] Figure 2 The present application is a behavior classification network data flow processing flowchart;

[0037] Figure 3This is a flow chart of the target tracking algorithm in the present invention;

[0038] Figure 4 It is a diagram of the overall process product of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0041] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0042] First, some terminology:

[0043] 1. Image semantics: refers to the meaning of image content. Image semantics consists of three parts: visual layer, object layer, and conceptual layer. The visual layer contains the underlying feature semantics, including color, shape, texture and other information; the object layer is equivalent to the middle layer, containing the attribute characteristics of the object, indicating its state at a certain moment; the conceptual layer belongs to the high-level layer, which is the image expression closest to human understanding, equivalent to the meaning expressed by this picture.

[0044] 2. Target tracking: In surveillance video, given the target position in the first frame, the target position in subsequent frames is predicted based on the tracking algorithm, and then associated and matched with the position in the previous frame, thereby achieving tracking of the constantly changing target.

[0045] 3. IOU: The full name is intersection over union, which is mainly used in target detection. It calculates the overlap rate of the "prediction box" and the "real box", that is, the ratio of their intersection and union. It is a standard used to measure the accuracy of detecting corresponding objects in a specific data set.

[0046] 4. Soft edge: The edge refers to the local discontinuous feature of the image, which is usually judged by the grayscale value. The location where the grayscale value in the image changes suddenly, that is, the location with a large gradient value is the edge. The soft edge refers to the gradient value at the edge that changes in a "gradual" manner rather than a rapid change.

[0047] Example 1

[0048] This embodiment discloses a method for constructing a scene-adaptive video dataset based on semantic analysis. Figure 1 As shown, the specific steps include:

[0049] Step 1: semantic analysis of target scene and existing scene, scene similarity matching, obtaining similar existing scene data of target scene.

[0050] Since some behaviors have specific occurrence scenes, in order to avoid constructing video data that does not match the behavior and scene, first, the target scene and the existing scene are subjected to image semantic analysis, the image content information under each scene is analyzed, including the environmental background of the scene, the object attribute features existing in the scene, and the similarity of the target scene and the existing scene is matched according to the semantic information, and the behavior data under the existing scene similar to the target scene is obtained.

[0051] Step 2: using the existing pre-trained behavior classification model to classify the video data in the similar scene, and obtaining the target behavior data.

[0052] The living body detection model R-C3D is adopted to realize the behavior recognition and classification of the existing behavior video data. The network model mainly consists of a shared 3D feature extraction network, a time suggestion subnetwork, and a behavior classification subnetwork. The 3D full convolution network is used to encode the video stream, and the features are calculated by convolution, which are shared by the subsequent time suggestion subnetwork and behavior classification subnetwork; the time suggestion subnetwork takes the features as input to generate different length time range proposals that may contain behaviors; the behavior classification subnetwork filters the proposals, uses the 3D pooling layer to convert the feature map output by the 3D convolution and the time range segment of different lengths into a fixed length feature map, and finally performs behavior category classification and predicts the refined behavior boundary. The data flow processing process of R-C3D is shown in Figure 2

[0053] The network is trained by optimizing the classification and regression tasks of the two subnetworks. In the training process, when judging the gap between the predicted behavior and the real behavior, the combination of two loss functions is used. The Softmax loss function is used for the judgment of the classification result, and the smooth L1 loss function is used for the regression. The specific formula is as follows:

[0054]

[0055] where N cls and N reg represent the size of the batch and the number of proposal segments respectively, λ is the loss weighting parameter, which takes the value of 1, t i represents the predicted relative offset of the proposals, represents the coordinate transformation of the ground truth of the proposals.

[0056] ​Step 3: The algorithm analyzes the target behavior video data, performs target tracking, and obtains continuous motion target sequence information.

[0057] The existing target behavior video data is input into the target tracking deep learning program, the detected motion targets are tracked, the detection boxes and coordinate positions of each motion target are obtained frame by frame or frame by frame, and the target detection boxes of each motion object are saved respectively to obtain a continuous behavior target sequence detection box image with time correlation information.

[0058] The above target tracking adopts the deep learning multi-target tracking ByteTrack method. First, target detection is performed on each frame of the video, and the association between the detection boxes of the detected targets in the front and back frames is performed to achieve the effect of motion object tracking. When tracking and associating the targets, not only the objects with high association scores between the detection boxes in the current frame and the detection boxes in the previous frame are considered, but also the targets with severe occlusion and low detection box association scores. At the same time, in order to obtain more accurate detection boxes, Kalman filtering is used to predict the detection boxes of the next frame, calculate the IOU relationship loss matrix between the predicted boxes and the targets with high association scores, and use the Hungarian algorithm to match the preliminary tracking trajectories and high-score boxes, and assign new IDs and trajectories to the high-score detection boxes that are not successfully matched; for low-score detection boxes, perform IOU matching with the above unsuccessfully matched tracking trajectories, and fuse the predicted boxes and the detection boxes to enhance the detection results of low-association-score detection boxes. The target tracking algorithm flow chart is shown in Figure 3 .

[0059] Kalman filtering is an algorithm that uses input observation data and the state of the previous period to perform optimal estimation of the state of the next period system through a linear state equation. The specific formula is as follows:

[0060]

[0061] where, represents the estimated value of the system state x k , z k represents the observation value of this period, g k is the Kalman gain, which is variable and updated once every period, between 0 and 1, representing the degree of trust in the measurement value.

[0062] Step 4: Perform background subtraction on the motion entities in the target sequence detection box image to obtain the behavior target sequence.

[0063] The foreground information and background information in each target sequence detection frame image obtained in step 2 are separated by using the BackgroundMatting algorithm, the image background is removed, and the image information of the behavior target sequence of interest with time correlation is obtained. The algorithm introduces the concept of an alpha (transparency) channel, and considers that the image is composed of foreground, background and transparency:

[0064] C p = a p F p + (1-a p )B p

[0065] Wherein, C p represents the color of the image, F p , B p , a p are unknown quantities, respectively representing foreground, background, and transparency, a p is a continuous value between 0 and 1, which can be understood as the probability that the pixel belongs to the foreground. Extracting the foreground image is to calculate the value of F p . In the process of use, the image or video and its corresponding background without people are input. First, the Adobe dataset is used to supervise the training of the network, and the processed original image and the auxiliary information generated by the adjacent frame are input into the network together with the background Figure One for training. The input data will first pass through an encoder, and after a series of iterative training, the network can ignore similar backgrounds and pay more attention to the feature segmentation of the portrait of interest. Then, through the residual structure and decoder module, a p and foreground F p are output, and the error loss of a p , F p and the original image color is continuously reduced. After several iterations of training, the foreground target sequence of interest is finally extracted.

[0066] Step 5: Scene adaptive conversion of target sequence according to attribute information of target scene graph.

[0067] Due to the differences in brightness, color, pixels and other environmental factors between different scenes, if the behavior target and the target background image are simply spliced without any processing, the problem of low fusion degree of foreground and background and the target being conspicuous will usually occur, which is due to the obvious change of the pixel position of the image. In order to avoid the above problems affecting the local convolution of the image when the behavior dataset is used for model training, the background image needs to be processed by an algorithm to calculate the basic attributes of the image, and the behavior target image needs to be adapted to the attributes, so as to obtain a target sequence with high fusion degree and high matching degree with the target background.

[0068] Step 6: Determine the region of interest in the target scene graph, and limit the activity range of the target sequence.

[0069] Due to the difference in the content of each scene image, including the diversity of object positions such as occlusion, road, table and chair, vehicle, wall, etc., the activity range of the behavior target in the current target background must be considered according to the unique characteristics of each scene. Write a program to divide the target scene image into regions and obtain the actual interesting activity region.

[0070] Step 7: Optimize the sequence edge to obtain a mixed image with "soft edge", and further obtain the video sequence of the target behavior in the target scene.

[0071] In order to further avoid the dramatic change of pixels in the target scene image operation position, and solve the "strong edge" problem, the SmoothMix strategy is adopted, and a smooth transition mask target G is generated based on the shape attribute of the target object. Usually, the center point coordinates (μ ω , μ h ) of the input target image and the diffusion σ in the image space are used to define G, μ ω and μ h are uniformly sampled in the width W and height H range of the input image, and σ defines the diffusion degree of the mask, which means that increasing its value will expand the range of the mask region. The total calculation formula is as follows:

[0072] μ ω ~ Unif(0, W), μ h ~ Unif(0, H)

[0073] The target sequence after a series of scene adaptive conversion is scaled according to the speed, relative position and other information in the original video sequence, and is spliced into the interesting region described in step 5 to obtain a series of mixed images with high fusion degree and "soft edge". These images are spliced by frame to generate video data of the behavior in the target scene.

[0074] The products of each process are shown in Figure 4 .

[0075] In summary, based on the various behavior video data of the existing scene, the target scene and the existing scene are analyzed and matched in terms of semantics, the pre-trained behavior classification model is used to classify the existing behavior data, the target behavior video data is obtained, the scene adaptive mechanism is proposed, the target tracking algorithm based on the content of the video sequence is designed, the behavior target sequence information is obtained, the background subtraction algorithm is used to extract the foreground target from the target sequence detection frame, and then the scene adaptive conversion, target sequence migration and optimization are performed according to the properties and content of the target scene, and high-quality video data of the target behavior in the target scene is generated.

[0076] The embodiment can realize the generation of a scene-unlimited behavior video data set according to a single target scene graph based on limited scene behavior video data, thereby providing a data basis for the construction of a behavior recognition model.

[0077] Embodiment two:

[0078] The scene-adaptive video data set construction system based on semantic analysis comprises:

[0079] The scene semantic analysis matching module is configured to perform semantic analysis on a target scene and an existing scene, obtain scene content attribute features, and perform scene similarity matching on the target scene and the existing scene to obtain existing scene behavior data similar to the target scene. The input is: target scene and existing scene images; and the output is: existing scene behavior data similar to the target scene.

[0080] The behavior video classification module under the existing scene is configured to input behavior video data of a similar scene under an existing scene into a behavior classification network model R-C3D for batch classification, and obtain classification labels of each behavior. The input is: behavior video data of a similar scene under an existing scene and a behavior classification model; and the output is: a classification result of the behavior data.

[0081] The target tracking and sequence information extraction module is configured to track a moving entity in a video based on target behavior video data under an existing scene by using a target tracking algorithm, and save coordinate, time relationship, detection frame and other information of the moving target. The input is: behavior video data under an existing scene; and the output is: behavior target sequence information.

[0082] The target sequence extraction module is configured to obtain required target behavior data according to requirements of a target scene, and use a Background Matting algorithm to remove the background of a target detection frame in each frame to obtain a behavior target sequence. The input is: a behavior target sequence detection frame image in behavior data; and the output is: a behavior target sequence.

[0083] The scene-adaptive conversion module is configured to analyze attributes of a target scene graph, perform scene-adaptive conversion on a behavior target sequence, and obtain a target sequence more similar to the target scene. The input is: a behavior target sequence; and the output is: a behavior target sequence after scene-adaptive conversion.

[0084] The target scene range limiting module is configured to use a program to analyze the content of a target scene graph, and limit the area where a target sequence actually exists. The input is: a target scene graph; and the output is: a target scene graph with a region of interest divided.

[0085] The target sequence edge optimization and synthesis module is configured to synthesize the target sequence with the target background image by frame using a SmoothMix strategy to obtain a synthesized image with a high fusion degree of a soft edge. The input is the behavior target sequence after scene adaptation and the target background image with the region of interest divided; and the output is a behavior synthesized image sequence with a high fusion degree.

[0086] The behavior video data generation module is configured to splice the behavior synthesized images by frame to obtain video data of a specific behavior in a target scene with continuous behavior actions. The input is the behavior synthesized image sequence; and the output is the specific behavior video data in the target scene.

[0087] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied therein.

[0088] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure One The functions specified in a flow or multiple flows and / or blocks Figure One The means for implementing the functions specified in a flow or multiple flows and / or blocks.

[0089] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure One The functions specified in a flow or multiple flows and / or blocks Figure One The means for implementing the functions specified in a flow or multiple flows and / or blocks.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams.Figure One one or more processes and / or functions specified in one or more blocks. Figure One one or more processes and / or functions specified in one or more blocks.

[0091] The above description is merely illustrative of the application, and is not intended to limit the scope of the application. Various modifications and changes can be made by those of ordinary skill in the art without departing from the spirit and scope of the application. Any modification, equivalent replacement, improvement, and the like made within the principle and technical scope of the application should be included in the protection scope of the application.

[0092] The above description is merely illustrative of the application, and is not intended to limit the scope of the application. Various modifications and changes can be made by those of ordinary skill in the art without departing from the spirit and scope of the application. Any modification, equivalent replacement, improvement, and the like made within the principle and technical scope of the application should be included in the protection scope of the application.

Claims

1. A method for constructing a scene-adaptive video dataset, characterized in that: The following steps are involved: Based on various behavioral video data of existing scenes, semantic analysis and similarity matching are performed between the target scene and the existing scenes; Obtain behavioral data of existing scenes whose similarity exceeds a predetermined value, classify them using a pre-trained behavior classification model, and obtain target behavior video data; Track the target behavior video data to obtain the behavior target sequence information, and extract the foreground target from the target sequence detection frame; According to the attributes and content of the target scene, the behavior target sequence is adaptively converted to obtain a target sequence that is more similar to the target scene; The target sequence and the target background image are synthesized and spliced ​​to obtain a video dataset of the required behavior in the target scene with continuous behavior actions; The specific process of extracting foreground targets from target sequence detection frames includes using background subtraction methods to separate the foreground information and background information in each target sequence detection frame image in each frame, removing the image background, and obtaining the temporally correlated behavioral target sequence image information of interest; Introducing the concept of transparency channel, ; in, Indicates the color of the image, 、 、 are all unknown quantities, representing the foreground, background, and transparency, respectively. It is a continuous value between 0 and 1, which is the probability that the pixel at that point belongs to the foreground. The processed original image and the auxiliary information generated by the adjacent frames are fed into the encoder together with the background image. Then, the transparency and foreground are output through the residual structure and decoder module, and the error between transparency, foreground and original image color is continuously reduced. After several iterative trainings, the foreground target sequence of interest is finally extracted.

2. The method for constructing a scene-adaptive video dataset according to claim 1, wherein: Based on various behavioral video data of existing scenes, the specific process of semantic analysis and similarity matching of the target scene and the existing scenes includes: semantic analysis of the target scene and the existing scenes, obtaining scene content attribute features, and scene similarity matching of the target scene and the existing scenes, and obtaining existing scene behavior data similar to the target scene.

3. The method for constructing a scene-adaptive video dataset according to claim 1, wherein: The specific process of classification using a pre-trained behavior classification model includes sending behavior video data of similar existing scenes into the behavior classification network model for batch classification, encoding the video stream in turn, calculating features through convolution, using the features as input, generating time ranges of different lengths that may contain behaviors, filtering the time ranges, converting the features and time range segments of different lengths into feature maps of fixed length, and finally classifying the behavior categories and predicting the refined behavior boundaries.

4. The method for constructing a scene-adaptive video dataset according to claim 1, wherein: The specific process of target tracking for target behavior video data includes: using the target tracking algorithm to track the moving entities in the video, obtaining the detection frame and coordinate position of each moving target frame by frame or every other frame, and saving the target detection frame of each moving object separately to obtain a continuous behavior target sequence detection frame image with time-related information.

5. The method for constructing a scene-adaptive video dataset according to claim 1, wherein: The specific process of synthesizing and splicing the target sequence with the target background image includes synthesizing the target sequence with the target background image frame by frame to obtain a synthesized image, and splicing the behavior synthesized image frame by frame to obtain video data of a specific behavior in a target scene with continuous behavior actions.

6. A system for constructing a scene-adaptive video dataset, characterized by: include: A scene semantic analysis and matching module is configured to perform semantic analysis and similarity matching between the target scene and the existing scene based on various behavior video data of the existing scene; The existing scene behavior video classification module is configured to obtain behavior data of existing scenes whose similarity exceeds a predetermined value, classify them using a pre-trained behavior classification model, and obtain target behavior video data; The target sequence information extraction module is configured to track the target behavior video data, obtain the behavior target sequence information, and extract the foreground target from the target sequence detection frame; A scene adaptive conversion module is configured to perform scene adaptive conversion on the behavior target sequence according to the attributes and content of the target scene to obtain a target sequence that is more similar to the target scene; A behavior video data generation module is configured to synthesize and splice the target sequence with the target background image to obtain a video dataset of the desired behavior in the target scene with continuous behavioral actions; The specific process of extracting foreground targets from target sequence detection frames includes using background subtraction methods to separate the foreground information and background information in each target sequence detection frame image in each frame, removing the image background, and obtaining the temporally correlated behavioral target sequence image information of interest; Introducing the concept of transparency channel, ; in, Indicates the color of the image, 、 、 are all unknown quantities, representing the foreground, background, and transparency, respectively. It is a continuous value between 0 and 1, which is the probability that the pixel at that point belongs to the foreground. The processed original image and the auxiliary information generated by the adjacent frames are fed into the encoder together with the background image. Then, the transparency and foreground are output through the residual structure and decoder module, and the error between transparency, foreground and original image color is continuously reduced. After several iterative trainings, the foreground target sequence of interest is finally extracted.

7. A computer-readable storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing the steps of any one of the methods of claims 1-5.

8. A terminal device, characterized in that: The method comprises a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executing the steps in the method according to any one of claims 1 to 5.

Images