Sample generation method and device, deep learning model training method and device, video stream processing method and device, equipment and medium

Abstract

The invention provides a sample generation method and device, a deep learning model training method and device, a video stream processing method and device, equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models, image processing and the like. The method comprises the steps of processing M candidate image sequences based on a first mode and a second mode respectively to obtain respective first detection information and second detection information of each candidate image sequence, the first detection information representing whether the candidate image sequence comprises at least one of a target object and a motion state of the target object, and the second detection information representing whether the candidate image sequence comprises at least one of the target object and the motion state of the target object; the second detection information represents whether the candidate image sequence comprises at least one of the target object and the motion state of the target object; determining N target image sequences according to the first detection information and the second detection information of each candidate image sequence; and generating a significant sample set according to the N target image sequences and the first detection information and the second detection information of each target image sequence.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, large models and image processing technology, and more specifically to a method and apparatus, device and medium for sample generation, deep learning model training and video stream processing. Background Art

[0002] With the development of computer technology, artificial intelligence technology has also been developed. For example, artificial intelligence technology can be used to process the massive production data generated by industrial parks, and monitor and analyze dangerous operating behaviors and abnormal environmental changes in the massive production data, in order to achieve the purpose of danger monitoring and alarm. Summary of the invention

[0003] The present invention provides a method, device, equipment and medium for sample generation, deep learning model training and video stream processing.

[0004] According to one aspect of the present disclosure, a sample generation method is provided, comprising: processing M candidate image sequences based on a first method and a second method respectively, to obtain first detection information and second detection information of each of the candidate image sequences, wherein the first detection information represents whether the candidate image sequence includes a target object and at least one of a motion state of the target object, and the second detection information represents whether the candidate image sequence includes the target object and at least one of the motion states of the target object, and M is a positive integer; determining N target image sequences in the M candidate image sequences according to the first detection information and the second detection information of each of the candidate image sequences, where N is a positive integer less than or equal to M; and generating a significant sample set according to the N target image sequences and the first detection information and the second detection information of each of the target image sequences.

[0005] According to one aspect of the present disclosure, a training method for a deep learning model is provided, comprising: obtaining a significant sample set, wherein the significant sample set is generated by using a sample generation method, and the significant sample set includes multiple target image sequences and label information of each of the target image sequences; inputting the target image sequence into a teacher model in a teacher-student network to obtain first sample detection information; inputting the target image sequence into a student model in the teacher-student network to obtain second sample detection information; and, according to the difference between the label information, the first sample detection information and the second sample detection information, adjusting the model parameters of the student model to obtain a trained deep learning model.

[0006] According to one aspect of the present disclosure, a video stream processing method is provided, including: obtaining a video stream to be processed; and inputting the above-mentioned video stream to be processed into a trained deep learning model to obtain a video stream processing result; wherein the above-mentioned trained deep learning model is trained using a deep learning model training method.

[0007] According to another aspect of the present disclosure, a sample generation device is provided, including: a first processing module, used to process M candidate image sequences based on a first method and a second method respectively, to obtain first detection information and second detection information of each of the above candidate image sequences, wherein the first detection information represents whether the above candidate image sequence includes a target object and at least one of the motion states of the target object, and the second detection information represents whether the above candidate image sequence includes the target object and at least one of the motion states of the target object, and M is a positive integer; a first determination module, used to determine N target image sequences in the above M candidate image sequences according to the first detection information and the second detection information of each of the above candidate image sequences, where N is a positive integer less than or equal to M; and a generation module, used to generate a significant sample set according to the above N target image sequences and the first detection information and the second detection information of each of the above target image sequences.

[0008] According to another aspect of the present disclosure, a training device for a deep learning model is provided, comprising: a first acquisition module, used to acquire a significant sample set, wherein the significant sample set is generated by using a sample generation device, and the significant sample set includes multiple target image sequences and label information of each of the target image sequences; a first input module, used to input the target image sequence into a teacher model in a teacher-student network to obtain first sample detection information; a second input module, used to input the target image sequence into a student model in the teacher-student network to obtain second sample detection information; and a training module, used to adjust model parameters of the student model according to the difference between the label information, the first sample detection information and the second sample detection information to obtain a trained deep learning model.

[0009] According to another aspect of the present disclosure, a video stream processing device is provided, including: a second acquisition module, used to acquire a video stream to be processed; and a third processing module, used to input the above-mentioned video stream to be processed into a trained deep learning model to obtain a video stream processing result; wherein the above-mentioned trained deep learning model is trained using a deep learning model training device.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, and the steps of the above method are implemented when the computer program or instructions are executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 The system architecture to which the sample generation, deep learning model training, and video stream processing method according to the embodiments of the present disclosure can be applied is schematically shown;

[0016] Figure 2 The flowchart of the sample generation method according to the embodiment of the present disclosure is schematically shown;

[0017] Figure 3 An example schematic diagram schematically shows a process of obtaining a candidate image sequence according to an embodiment of the present disclosure;

[0018] Figure 4 An example schematic diagram schematically shows a process of determining a target image sequence according to an embodiment of the present disclosure;

[0019] Figure 5 An example schematic diagram schematically shows a process of determining a target image sequence according to another embodiment of the present disclosure;

[0020] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 7 An example schematic diagram schematically illustrates a training process of a deep learning model according to an embodiment of the present disclosure;

[0022] Figure 8 The flowchart of the video stream processing method according to the embodiment of the present disclosure is schematically shown;

[0023] Fig. 9 A block diagram of a sample generating device according to an embodiment of the present disclosure is schematically shown;

[0024] Fig.10 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown;

[0025] Fig.11 A block diagram schematically shows a video stream processing device according to an embodiment of the present disclosure; and

[0026] Fig.12 The block diagram schematically shows an electronic device suitable for implementing sample generation, deep learning model training, and video stream processing methods according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] Industrial parks often generate massive amounts of production data, and the data carriers are mostly videos, so there will be a large number of highly repetitive and meaningless samples.

[0032] In one example, a deep learning model can be used to screen the entire amount of production data to obtain valid samples. For example, the video stream can be processed by extracting frames, and then the hash algorithm can be used to deduplicate the video frames at the graph level. The pre-trained deep learning model can be used to perform graph-level reasoning and prediction on the video frames, and then the possible valid samples can be screened out based on the prediction results. Finally, the sample set can be expanded by manually reviewing the data to achieve the purpose of iterative optimization of the model.

[0033] However, since the use of deep learning models to screen the entire amount of production data requires a lot of time and computing power, and a large amount of manpower is required to review the data, it is contrary to the tight pace of actual industrial production.

[0034] In addition, due to the low frequency of business data in some production safety scenarios and the difficulty in data collection, deep learning models often cannot obtain ideal detection capabilities in the cold start phase with less data. Traditional deep learning models can often only process data according to image frames and have difficulty understanding the semantic information of video streams in time sequence. Therefore, it is difficult to use deep learning models to accurately and efficiently extract real and valid samples from production data.

[0035] In addition, using deep learning models for data screening can often only obtain data that the model itself has a certain detection ability, but it is difficult to mine samples that are helpful for subsequent iterative optimization of the model. Since a large number of directionless and non-targeted samples are of no benefit to the training of deep learning models, it will not only increase the time consumption of model training, but may even cause the model to be optimized in the opposite direction.

[0036] Therefore, how to achieve simple, efficient and highly accurate effective video sample mining is an urgent problem to be solved.

[0037] To this end, the embodiment of the present disclosure proposes a sample generation scheme. For example, based on the first method and the second method, respectively, M candidate image sequences are processed to obtain the first detection information and the second detection information of each candidate image sequence, wherein the first detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and the second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and M is a positive integer; according to the first detection information and the second detection information of each candidate image sequence, N target image sequences are determined in the M candidate image sequences, where N is a positive integer less than or equal to M; and according to the N target image sequences and the first detection information and the second detection information of each target image sequence, a significant sample set is generated.

[0038] According to the embodiments of the present disclosure, since the first detection information and the second detection information are obtained by processing the candidate image sequence in different ways, the accuracy of the target image sequence screening can be improved by comparing the first detection information and the second detection information of each candidate image sequence and determining the target image sequence according to the difference between the two. On this basis, by generating a significant sample set according to the first detection information and the second detection information of each target image sequence, effective mining of significant samples can be achieved, and the quality and accuracy of the significant sample set can be improved.

[0039] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0040] In the technical solution of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0041] Figure 1 The schematic diagram shows the system architecture that can be applied to sample generation, deep learning model training, and video stream processing methods according to the embodiments of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0042] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0044] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0045] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0046] It should be noted that the sample generation, deep learning model training, and video stream processing methods provided in the embodiments of the present disclosure can generally be performed by the server 105. Accordingly, the sample generation, deep learning model training, and video stream processing apparatus provided in the embodiments of the present disclosure can generally be set in the server 105. The sample generation, deep learning model training, and video stream processing methods provided in the embodiments of the present disclosure can also be performed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the sample generation, deep learning model training, and video stream processing apparatus provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0047] Alternatively, the sample generation, deep learning model training, and video stream processing methods provided in the embodiments of the present disclosure may also be performed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the sample generation, deep learning model training, and video stream processing apparatus provided in the embodiments of the present disclosure may also be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be set in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0049] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0050] The above describes the system architecture that can be used for sample generation, deep learning model training, and video stream processing provided by the present disclosure. Figure 2 For example, the sample generation process of the present disclosure is further explained.

[0051] Figure 2 The flowchart of the sample generation method according to the embodiment of the present disclosure is schematically shown.

[0052] like Figure 2 As shown, the sample generation method 200 includes operations S210 to S230.

[0053] In operation S210, M candidate image sequences are processed based on the first method and the second method respectively to obtain first detection information and second detection information of each candidate image sequence, wherein the first detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and the second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and M is a positive integer.

[0054] In operation S220 , N target image sequences are determined from the M candidate image sequences according to the first detection information and the second detection information of each candidate image sequence, where N is a positive integer less than or equal to M.

[0055] In operation S230 , a salient sample set is generated according to the N target image sequences, the first detection information and the second detection information of each target image sequence.

[0056] A candidate image sequence refers to a set of images that meet requirements or specific conditions selected from a group of images in the field of computer vision and image processing. The multiple images in the image set have a temporal relationship, that is, the multiple images in the candidate image sequence are continuous in time. The candidate image sequence can be captured from the original video stream to be detected in the industrial park. The method of capturing the candidate image sequence can be configured according to actual business needs and is not limited here. For example, the method of capturing the candidate image sequence can include at least one of the following: using video editing software to capture the candidate image sequence, using a command line tool to capture the candidate image sequence, and using a programming language combined with a corresponding library to capture the candidate image sequence.

[0057] After obtaining the candidate image sequence, the candidate image sequence can be processed based on a first method to obtain first detection information. The first detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object. In addition, the candidate image sequence can be processed based on a second method to obtain second detection information. The second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object. The target object refers to the object that needs to be detected, tracked or identified in the candidate image sequence, that is, the key subject to be found. The motion state refers to the movement of the target object in the candidate image sequence. For example, the motion state may include one of the following: stop, move, rotate, etc.

[0058] The first detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, which means one of the following situations: the first detection information represents that the candidate image sequence does not include the target object; the first detection information represents that the candidate image sequence includes the target object; the first detection information represents that the candidate image sequence includes the target object and the motion state of the target object. The second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, which means one of the following situations: the second detection information represents that the candidate image sequence does not include the target object; the second detection information represents that the candidate image sequence includes the target object; the second detection information represents that the candidate image sequence includes the target object and the motion state of the target object.

[0059] The first method and the second method can be configured according to actual business needs, and are not limited here, as long as the candidate image sequence can be processed to obtain detection information and the two are different. For example, the first method and the second method may include at least one of the following: a method based on a traditional machine learning model, a method based on a deep learning model, a method based on key point detection, a method based on a large model, a method based on trajectory analysis and optical flow estimation, etc.

[0060] The method based on the traditional machine learning model refers to the use of the traditional machine learning model to extract and process the features of the candidate image sequence. The method based on the traditional machine learning model may include at least one of the following: a method based on a support vector machine (SVM), a method based on a decision tree (DT), and a method based on a random forest (RF). The method based on the deep learning model refers to the use of the deep learning model to extract and process the features of the candidate image sequence. The method based on the deep learning model may include at least one of the following: a method based on a recurrent neural network (RNN) and a method based on a long short-term memory network (LSTM).

[0061] Methods based on key point detection refer to using key points identified in each image included in the candidate image sequence to track the target object, analyze motion, or perform other types of sequence processing. Methods based on large models refer to using pre-trained large-scale deep learning models to process image sequences. Methods based on large models may include at least one of the following: methods based on artificial intelligence generative models (AIG) and methods based on large-scale generative adversarial networks (BigGAN). Methods based on trajectory analysis and optical flow estimation refer to analyzing the trajectory and motion of the target object in the candidate image sequence.

[0062] After obtaining the first detection information and the second detection information of each candidate image sequence respectively based on different methods, for each candidate image sequence, the first detection information and the second detection information can be compared, and the candidate image sequence whose first detection information and the second detection information are inconsistent is determined as a target image sequence. The target image sequence refers to an image sequence that can provide enough effective information for the content concerned by the task.

[0063] The specific determination criteria for the inconsistency between the first detection information and the second detection information are configured according to actual business needs and are not limited here. For example, the inconsistency between the first detection information and the second detection information may include at least one of the following situations: the first detection information includes the target object, and the second detection information does not include the target object; the first detection information does not include the target object, and the second detection information includes the target object; the first detection information and the second detection information both include the target object, but the target objects included in the two are different; the first detection information and the second detection information both include the target object, but the motion states of the target objects included in the two are different, etc.

[0064] After obtaining the target image sequence, for each target image sequence, the detection information that can correctly characterize the target object and the motion state of the target object in the target image sequence can be determined in the first detection information and the second detection information, and used as the label information corresponding to the target image sequence. On this basis, a significant sample set can be generated according to the target image sequence and the label information of each target image sequence. The significant sample set includes multiple significant samples. A significant sample refers to a sample that can provide enough effective information for the content of the task. For example, if the task is to identify dogs, samples containing dogs are high-value samples, while samples containing only cats are not high-value samples.

[0065] According to the embodiments of the present disclosure, since the first detection information and the second detection information are obtained by processing the candidate image sequence in different ways, the accuracy of the target image sequence screening can be improved by comparing the first detection information and the second detection information of each candidate image sequence and determining the target image sequence according to the difference between the two. On this basis, by generating a significant sample set according to the first detection information and the second detection information of each target image sequence, effective mining of significant samples can be achieved, and the quality and accuracy of the significant sample set can be improved.

[0066] In the embodiment of the present disclosure, the candidate image sequence in the sample generation method can be obtained by intercepting the video stream to be detected. Figure 3 The process of obtaining a candidate image sequence is described.

[0067] Figure 3 An example schematic diagram of a process for obtaining a candidate image sequence according to an embodiment of the present disclosure is schematically shown.

[0068] like Figure 3 As shown, in 300, the video stream 301 to be detected may be obtained by monitoring an industrial park. The video stream 301 to be detected may include video data and audio data transmitted in a stream form in a time sequence.

[0069] After receiving the video stream 301 to be detected, for each video frame 302 in the video stream 301 to be detected, the video frame 302 can be processed to obtain a mask image 303. For example, background modeling and foreground detection processing can be performed on the video frame 302, and the mask image 303 can be obtained according to the difference between the foreground object and the background image. The mask image 303 can be used to characterize which pixels belong to the foreground and which pixels belong to the background for the video frame 302, that is, the mask image 303 can effectively show the part of interest in the video frame 302. The method for obtaining the mask image 303 may include at least one of the following: a method based on a Gaussian Mixture Model (GMM), a frame difference method, or a method based on deep learning.

[0070] In one example, after obtaining the mask image 303, the mask image 303 may be post-processed to improve the quality of the mask image 303. The post-processing operation may include one of the following: filling, connectivity analysis, and morphological processing. For example, the mask image 303 may be corroded using a preset convolution kernel to weaken or segment the peninsula-type connection in the mask image 303, that is, the foreground area of ​​the pixels in the mask image 303 may be gradually reduced using a preset convolution kernel to remove the noise of the mask image 303, so that different areas are independent of each other, and a better segmentation effect is achieved. On this basis, the mask image 303 may also be expanded using the preset convolution kernel to fill and obtain a foreground contour. The size of the preset convolution kernel may be 3*3.

[0071] The pixel value of the foreground part in the mask image 303 is a first preset value, and the pixel value of the background part is a second preset value, the first preset value may be 1, and the second preset value may be 0. After the mask image 303 is obtained, the grayscale values ​​of the adjacent video frames at the same position may be compared pixel by pixel based on the respective binary images 304 of the mask images 303 of the adjacent video frames, and the difference between the grayscale values ​​of the adjacent video frames at the same position may be determined to obtain the target difference 305, so as to compare the changes of the mask images 303 of the previous and next frames.

[0072] The target difference 305 can be used to screen video samples that are suspected of having moving objects or abnormal changes in the environment between the previous and next frames. After the target difference 305 is obtained, operation S310 can be performed.

[0073] In operation S310, it is determined whether the target difference 305 satisfies a predetermined condition. The predetermined condition can be configured according to actual business needs and is not limited here. For example, the predetermined condition can be that the target difference 305 is greater than or equal to a predetermined threshold.

[0074] If so, the candidate image sequence 306 can be extracted from the video stream 301 to be detected according to the timestamp of the video frame related to the video frame 302. The timestamp can indicate the start time and the end time of the video frame 302, so multiple continuous video frames 302 can be located in the video stream 301 to be detected according to the timestamp of the video frame related to the video frame 302, and these continuous video frames can be extracted from the video stream 301 to be detected to obtain the candidate image sequence 306. The candidate image sequence 306 is an image sequence obtained through preliminary screening that may contain target objects in motion. Compared with a static image sequence, the candidate image sequence 306 is more likely to contain dangerous operating behaviors or abnormal environmental changes.

[0075] According to the embodiments of the present disclosure, since the mask image is obtained by processing the video frame, the difference between the grayscale values ​​at the same position is determined by comparing the mask images of the adjacent video frames, and the timestamp is determined according to the target difference and the predetermined condition, so as to intercept the candidate image sequence from the video stream to be detected based on the timestamp, thereby being able to effectively identify and intercept the required content. In addition, since it is possible to quickly filter the image sequence without target objects or abnormal environmental changes, the image sequence suspected of having target objects in motion or abnormal environmental changes is screened out, thereby reducing the amount of screened data, compressing the data processing time, and being able to recall as many possible significant samples as possible while reducing the amount of data processing in the subsequent steps, thereby reducing the computing power and time costs for the subsequent model inference stage.

[0076] In the embodiment of the present disclosure, the target image sequence can be determined by performing a coarse-grained comparison between the first detection information and the second detection information of each candidate image sequence. Figure 4 The method of determining the target image sequence is described.

[0077] Figure 4 An example schematic diagram schematically illustrates a process of determining a target image sequence according to an embodiment of the present disclosure.

[0078] like Figure 4 As shown, in 400, the first method can be a method based on a large model, and the second method can be a method based on a deep learning model. In this case, the trained large model 402 and the trained deep learning model 404 can constitute a teacher-student network, the trained large model 402 with strong detection and generalization capabilities serves as the teacher model in the teacher-student network, and the trained deep learning model 404 serves as the student model in the teacher-student network. The trained deep learning model 404 can be obtained by training using the trained large model 402. The teacher model can provide knowledge and guidance to the student model, and the student model can introduce the instructions of the teacher model and has a lightweight structure.

[0079] Processing the candidate image sequence based on the first method may refer to inputting the candidate image sequence 401 into the trained large model 402 to obtain the first detection information 403. Processing the candidate image sequence based on the second method may refer to inputting the candidate image sequence 401 into the trained deep learning model 404 to obtain the initial detection information 405. The initial detection information 405 characterizes whether the candidate image sequence 401 includes the target object. On this basis, the target tracking model 406 may be used to track the target of the video frames of the multiple consecutive timestamps included in the candidate image sequence 401 according to the initial detection information 405 to obtain the second detection information 407.

[0080] According to the embodiments of the present disclosure, by introducing a teacher-student network mechanism, that is, using a trained large model to train a trained deep learning model, the large model with stronger detection capability is used as the teacher model, and the deep learning model is used as the student model, so as to make use of the characteristics of high detection capability and strong generalization capability that the large model can obtain based on a small amount of data, and use the two models to analyze and process the candidate image sequences separately at the image sequence level, so as to achieve the purpose of detecting the target object and the motion state of the target object, and provide a basis for subsequent detection information comparison.

[0081] After obtaining the first detection information 403 and the second detection information 407, significant samples that are helpful for the iteration of the teacher model and the student model can be selected by horizontally comparing the differences in the prediction results of the teacher model and the student model.

[0082] In one example, the difference comparison of the prediction results of the teacher model and the student model may include a coarse-grained comparison process. For example, operation S410 may be performed. In operation S410, it is determined whether the first detection information 403 and the second detection information 407 are consistent.

[0083] If so, the candidate image sequence 401 can be deleted. It should be noted that the first detection information 403 and the second detection information 407 are consistent and may include at least one of the following situations: the first detection information 403 and the second detection information 407 do not include the target object; the first detection information 403 and the second detection information 407 include the same target object, and the motion state of the target object included in both is the same.

[0084] If not, the candidate image sequence 401 may be determined as the target image sequence 408. It should be noted that the inconsistency between the first detection information 403 and the second detection information 407 may include at least one of the following situations: the first detection information 403 includes the target object, and the second detection information 407 does not include the target object; the first detection information 403 does not include the target object, and the second detection information 407 includes the target object; the first detection information 403 and the second detection information 407 both include the target object, but the target objects included in the two are different; the first detection information 403 and the second detection information 407 both include the target object, but the motion states of the target objects included in the two are different.

[0085] According to the embodiments of the present disclosure, by comparing and verifying the first detection information output by the trained large model and the second detection information output by the deep learning model, the candidate image sequences with poor detection effect of the deep learning model are selected by taking advantage of the characteristics of the large model's high detection ability and strong generalization ability, so that more target image sequences that can serve as significant samples can be obtained during the secondary data screening, and the obtained target image sequences can guide the optimization iteration of the deep learning model in a more directional and targeted manner, which is beneficial to improving the quality of the significant sample set, so that the deep learning model can be optimized in the desired direction at the accelerated speed in the subsequent model training stage.

[0086] In the embodiment of the present disclosure, the target image sequence can also be determined by performing a coarse-grained comparison and a fine-grained comparison on the first detection information and the second detection information of each candidate image sequence. Figure 5 The method of determining the target image sequence is described.

[0087] Figure 5 An example schematic diagram of a process for determining a target image sequence according to another embodiment of the present disclosure is schematically shown.

[0088] like Figure 5 As shown, in 500, the first method can be a method based on a large model, and the second method can be a method based on a deep learning model. In this case, the trained large model 502 and the trained deep learning model 504 can also constitute a teacher-student network.

[0089] Processing the candidate image sequence based on the first method may refer to inputting the candidate image sequence 501 into the trained large model 502 to obtain first detection information 503. Processing the candidate image sequence based on the second method may refer to inputting the candidate image sequence 501 into the trained deep learning model 504 to obtain initial detection information 505. The initial detection information 505 characterizes whether the candidate image sequence 501 includes the target object. On this basis, the target tracking model 506 may be used to track the target of the video frames of the multiple consecutive timestamps included in the candidate image sequence 501 according to the initial detection information 505 to obtain second detection information 507.

[0090] After obtaining the first detection information 503 and the second detection information 507, significant samples that are helpful for the iteration of the teacher model and the student model can be selectively selected by horizontally comparing the differences in the prediction results of the teacher model and the student model.

[0091] In one example, the difference comparison of the prediction results of the teacher model and the student model may include a coarse-grained comparison process and a fine-grained comparison process. For example, operation S510 may be performed. In operation S510, it is determined whether the first detection information 503 and the second detection information 507 are consistent? This operation S510 is equivalent to a coarse-grained comparison process.

[0092] If so, the candidate image sequence 501 may be deleted. The case where the first detection information 503 and the second detection information 507 are consistent will not be described in detail here.

[0093] If not, the first difference 508 can be determined according to the first confidence level related to the target object included in both the first detection information 503 and the second detection information 507, and the second difference 509 can be determined according to the second confidence level related to the motion state included in both the first detection information 503 and the second detection information 507. The first confidence level refers to the degree of possibility or certainty of whether the target object exists or not. The second confidence level refers to the degree of possibility or certainty of the motion state of the target object.

[0094] After obtaining the first difference 508 and the second difference 509, operation S520 may be performed. In operation S520, it may be determined whether the first difference 508 and the second difference 509 are both greater than or equal to a predetermined confidence threshold. This operation S520 is equivalent to a fine-grained comparison process. The predetermined confidence threshold may be configured according to actual business needs and is not limited here. For example, the higher the predetermined confidence threshold, the stricter the determination of the target image sequence 510. If not, the candidate image sequence 501 may be deleted.

[0095] If so, since samples with different prediction results between the teacher model and the student model often represent that one of the models has room for improvement in its recognition ability for the candidate image sequence, the candidate image sequence 501 can be determined as the target image sequence 510 to achieve targeted screening of effective significant samples, thereby better guiding the model to optimize in a specified direction. The target image sequence 510 refers to a candidate image sequence in which there is a difference between the first detection information 503 output by the trained large model 502 and the second detection information 507 output by the trained deep learning model 504.

[0096] According to the embodiments of the present disclosure, the first prediction result is obtained by inputting the initially screened candidate video samples into the trained large model, and the second prediction result is obtained by inputting the initially screened candidate video samples into the trained deep model. On this basis, in the process of comparing the differences between the two, coarse-grained comparison and fine-grained comparison are introduced, that is, not only comparing whether the two are consistent, but also comparing the relationship between the confidence of the two and the predetermined confidence threshold, so that the accuracy of data screening can be improved while reducing the cost of manual review processing, and effective video samples that are beneficial to the iteration of deep learning small models can be screened out in a targeted manner, avoiding the reduction of model performance caused by disordered and aimless data supplementation. In addition, by forming a virtuous cycle of significant sample generation and model iterative training, a high accuracy rate of directional data screening is achieved, which can accelerate the optimization of deep learning small models in the desired direction, and lay a solid data foundation for the subsequent improvement of deep learning model effects.

[0097] In one example, after obtaining the target image sequence, a significant sample set can be generated according to N target image sequences, the first detection information of each target image sequence and the second detection information. The significant sample set includes multiple significant samples. A significant sample refers to a sample that can provide enough effective information for the content of the task. Each significant sample may include a target image sequence and label information. The label information can be determined based on the first detection information and the second detection information, and can correctly characterize whether there is a target object in the target image sequence and the motion state of the target object when the target object exists.

[0098] The method for determining the tag information can be configured according to actual business needs and is not limited here. For example, the method for determining the tag information can include at least one of the following: a method based on manual review, a method based on ensemble learning, and a method based on confidence.

[0099] The method based on manual review refers to displaying the target image sequence, the first detection information and the second detection information to the user, and the user determines the correct label information. The method based on ensemble learning refers to inputting the first detection information and the second detection information into an ensemble learning algorithm, such as a voting classifier or a random forest, to comprehensively consider the first detection information and the second detection information to determine the final label information. The method based on confidence refers to determining the information with higher confidence in the first detection information and the second detection information as the label information.

[0100] For example, a first weight corresponding to the confidence of the target object and a second weight corresponding to the motion state of the target object may be pre-set. For the first detection information, a first product value may be determined based on the first weight and the first confidence included in the first detection information; a second product value may be determined based on the second weight and the second confidence included in the first detection information; and a first cumulative value may be determined based on the first product value and the second product value. For the second detection information, a third product value may be determined based on the first weight and the first confidence included in the second detection information; a fourth product value may be determined based on the second weight and the second confidence included in the second detection information; and a second cumulative value may be determined based on the third product value and the fourth product value. On this basis, the first cumulative value and the second cumulative value may be compared, and the detection information corresponding to the larger cumulative value may be determined as label information.

[0101] After obtaining the label information of the target image sequence, the salient samples can be determined according to the target image sequence and the label information, and the salient samples can be appended to the salient sample set to expand the salient sample set, so as to facilitate the subsequent use of the salient sample set to iteratively train large models and deep learning models, thereby forming a closed loop of the entire process of salient sample generation and model iterative training.

[0102] In one example, the significant sample set may include a training sample set and a qualified sample set. The training sample set can be understood as a training set and a validation set, which are used to guide the subsequent training process of the deep learning model. The qualified sample set can be understood as a test set, which is used to test the model performance of the trained deep learning model, and when the performance test results of the trained deep learning model meet the predetermined performance conditions, the trained deep learning model is qualified to go online and put into the next round of sample screening.

[0103] According to an embodiment of the present disclosure, pre-annotated label information is obtained based on the first detection information and the second detection information of the target image sequence, and then a significant sample set is generated based on the target image sequence and the label information. This can reduce a large amount of invalid and repeated data in the original video stream to be detected, screen out effective significant samples, and improve the accuracy of data screening while reducing the cost of manual review processing. This ensures that subsequent models can be iteratively optimized in a more targeted manner, which is conducive to improving the training effect of the machine learning model, thereby better meeting the needs of industrial safety production scenarios.

[0104] The above are only exemplary embodiments, but are not limited thereto, and may also include other sample generation methods known in the art, as long as the quality and accuracy of the significant sample set can be improved.

[0105] The above describes the sample generation method of the present disclosure. After generating a significant sample set using the sample generation method, the significant sample set can also be used to train a deep learning model to obtain a trained deep learning model. Figure 6 This section explains how to train a deep learning model.

[0106] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0107] like Figure 6 As shown, the deep learning model training method 600 includes operations S610~S640.

[0108] In operation S610, a salient sample set is obtained, wherein the salient sample set is generated by using a sample generation method, and the salient sample set includes a plurality of target image sequences and label information of each target image sequence.

[0109] In operation S660, the target image sequence is input into a teacher model in a teacher-student network to obtain first sample detection information.

[0110] In operation S630, the target image sequence is input into a student model in the teacher-student network to obtain second sample detection information.

[0111] In operation S640, model parameters of the student model are adjusted according to the difference between the label information, the first sample detection information, and the second sample detection information to obtain a trained deep learning model.

[0112] The tag information can be used to characterize whether the target image sequence includes at least one of the actual target object and the actual motion state of the target object. The target object refers to the object that needs to be detected, tracked or identified in the target image sequence, that is, the key subject to be found. The motion state refers to the motion of the target object in the target image sequence. For example, the motion state can include one of the following: stop, move, rotate, etc.

[0113] The label information represents whether the target image sequence includes at least one of the actual target object and the actual motion state of the target object, which refers to one of the following situations: the label information represents that the target image sequence does not include the actual target object; the label information represents that the candidate image sequence includes the actual target object; the label information represents that the target image sequence includes the actual target object and the motion state of the actual target object.

[0114] The teacher-student network may include a teacher model and a student model. The teacher model refers to a model that obtains high performance after being trained with a large amount of data in the field of object detection. The student model refers to an object detection model that is more lightweight and simpler than the teacher model. The specific model selection of the teacher model and the student model can be configured according to actual business needs and is not limited here. For example, the teacher model and the student model may include at least one of the following: a traditional machine learning model, a deep learning model, a key point detection model, a large model, a trajectory analysis and optical flow estimation model, etc. In one example, the teacher model may be a large model, and the student model may be a deep learning model. The deep learning model may include one of the following: a supervised model, a semi-supervised model, and an unsupervised model.

[0115] After obtaining the salient sample set, for each target image sequence in the salient sample set, the target image sequence can be input into the teacher model and the student model respectively to obtain first sample detection information and second sample detection information. The first sample detection information represents whether the target image sequence includes at least one of the detected target object and the detected motion state of the target object. The second sample detection information represents whether the target image sequence includes at least one of the detected target object and the detected motion state of the target object.

[0116] After obtaining the first sample detection information and the second sample detection information, the label information, the first sample detection information and the second sample detection information can be input into the loss function to obtain the loss function value. The model parameters of the student model are adjusted according to the loss function value until a predetermined end condition is met. The student model obtained when the predetermined end condition is met is determined as a trained deep learning model. The predetermined end condition may include that the model iteration meets a predetermined number of times or the loss function converges.

[0117] According to the embodiments of the present disclosure, by using the significant sample set as training samples to obtain a trained deep learning model, it is possible to guide the iterative training process in a direction, reduce the number of model iterations, and improve the training speed and prediction accuracy of the model. Thus, the cost of training the deep learning model is reduced, so that the trained deep learning model can be applied to different video stream processing tasks, which helps to improve the accuracy and effect of subsequent video stream processing.

[0118] Figure 7 An example schematic diagram of the training process of a deep learning model according to an embodiment of the present disclosure is schematically shown.

[0119] like Figure 7 As shown in 700 , the salient sample set includes a plurality of salient samples 701 , and each salient sample 701 includes a target image sequence 701_1 and label information 701_2 .

[0120] For each significant sample 701, the target image sequence 701_1 can be input into the teacher model 702 in the teacher-student network to obtain first sample detection information 703. The target image sequence 701_1 can be input into the student model 704 in the teacher-student network to obtain second sample detection information 705.

[0121] After obtaining the first sample detection information 703 and the second sample detection information 705, the difference 706 between the label information 701_2, the first sample detection information 703 and the second sample detection information 705 can be determined. On this basis, the model parameters of the student model 704 can be adjusted according to the difference 706 to obtain a trained deep learning model.

[0122] The above are only exemplary embodiments, but are not limited thereto and may also include other deep learning model training methods known in the art, as long as they can improve the training speed and prediction accuracy of the model.

[0123] The above describes the training method of the deep learning model disclosed in the present invention. After the deep learning model is trained using the salient sample set and the trained deep learning model is obtained, the trained deep learning model can also be used to process the video stream to be processed. Figure 8 A video stream processing method is described.

[0124] Figure 8 The flowchart of the video stream processing method according to the embodiment of the present disclosure is schematically shown.

[0125] like Figure 8 As shown, the video stream processing method 800 includes operations S810-S820.

[0126] In operation S810, a video stream to be processed is obtained.

[0127] In operation S820, the video stream to be processed is input into a trained deep learning model to obtain a video stream processing result.

[0128] The video stream to be processed may be obtained by real-time monitoring of the industrial park. The video stream to be processed may include video data and audio data transmitted in a stream in chronological order. After obtaining the video stream to be processed, the video data and audio data in the video stream to be processed may be preprocessed to obtain an intermediate processed video stream.

[0129] For example, for video data, the video data may be decoded based on a preset video codec format. The preset video codec format may include, for example, H.264 and H.265. Alternatively, for audio data, the audio data may be decoded based on a preset audio codec format. The preset audio codec format may include, for example, AAC and MP3.

[0130] After obtaining the intermediate processed video stream, the intermediate processed video stream can be input into the trained deep learning model to obtain a video stream processing result. The trained deep learning model is trained using a deep learning model training method. The video stream processing result can be used to characterize whether the video stream to be processed includes the target object, and the motion state of the target object when the video stream to be processed includes the target object.

[0131] According to the embodiments of the present disclosure, when a trained deep learning model is used to process a video stream to be processed, video stream processing can be achieved more accurately, thereby improving the accuracy of the video stream processing results and the processing efficiency of the video stream to be processed.

[0132] The above are only exemplary embodiments, but are not limited thereto, and may also include other video stream processing methods known in the art, as long as they can improve the accuracy of the video stream processing results and the processing efficiency of the video stream to be processed.

[0133] Fig. 9 The block diagram of a sample generating device according to an embodiment of the present disclosure is schematically shown.

[0134] like Fig. 9 As shown, the sample generating device 900 may include a first processing module 910 , a first determining module 920 and a generating module 930 .

[0135] The first processing module 910 is used to process M candidate image sequences based on the first method and the second method respectively to obtain first detection information and second detection information of each candidate image sequence, wherein the first detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and the second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and M is a positive integer.

[0136] The first determination module 920 is configured to determine N target image sequences from M candidate image sequences according to the first detection information and the second detection information of each candidate image sequence, where N is a positive integer less than or equal to M.

[0137] The generating module 930 is used to generate a salient sample set according to the N target image sequences, the first detection information and the second detection information of each target image sequence.

[0138] According to an embodiment of the present disclosure, the first processing module 910 may include a first input unit, a second input unit, and a target tracking unit.

[0139] The first input unit is used to input each candidate image sequence into the trained large model to obtain first detection information.

[0140] The second input unit is used to input each candidate image sequence into the trained deep learning model to obtain initial detection information, wherein the initial detection information represents whether the candidate image sequence includes the target object. The trained deep learning model is obtained by training using the trained large model.

[0141] The target tracking unit is used to perform target tracking on a plurality of video frames with consecutive time stamps included in the candidate image sequence according to the initial detection information to obtain second detection information.

[0142] According to an embodiment of the present disclosure, the first determining module 920 may include a deleting unit and a first determining unit.

[0143] The deleting unit is used to delete the candidate image sequence for each candidate image sequence when the first detection information and the second detection information are consistent.

[0144] The first determining unit is configured to determine the candidate image sequence as the target image sequence when the first detection information and the second detection information are inconsistent.

[0145] According to an embodiment of the present disclosure, the first detection information and the second detection information each further include a first confidence level associated with the target object and a second confidence level associated with the motion state.

[0146] According to an embodiment of the present disclosure, the first determining unit may include a first determining subunit and a second determining subunit.

[0147] The first determining subunit is used to determine a first difference between the first confidences and a second difference between the second confidences.

[0148] The second determining subunit is configured to determine the candidate image sequence as the target image sequence in response to the first difference and the second difference being both greater than or equal to a predetermined confidence threshold.

[0149] According to an embodiment of the present disclosure, the generating module 930 may include a second determining unit and a generating unit.

[0150] The second determination unit is configured to determine label information for each target image sequence according to the first detection information and the second detection information, wherein the label information is used to identify whether the target image sequence includes at least one of a target object and a motion state of the target object.

[0151] The generating unit is used to generate a significant sample set according to each target image sequence and the label information of each target image sequence.

[0152] According to an embodiment of the present disclosure, the sample generating device 900 may further include a second processing module, a second determining module and an intercepting module.

[0153] The second processing module is used for processing each video frame in the video stream to be detected respectively in response to receiving the video stream to be detected, so as to obtain a mask image of each video frame.

[0154] The second determination module is used to determine the difference between the grayscale values ​​of adjacent video frames at the same position based on the binary images of the mask images of the adjacent video frames to obtain a target difference.

[0155] The interception module is used to intercept a candidate image sequence from the video stream to be detected according to the time stamp of the relevant video frame of the video frame in response to the target difference satisfying the predetermined condition.

[0156] Fig.10 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0157] like Fig.10 As shown, the training device 1000 for the deep learning model may include a first acquisition module 1010, a first input module 1020, a second input module 1030 and a training module 1040.

[0158] The first acquisition module 1010 is used to acquire a salient sample set, wherein the salient sample set is generated by using the sample generation device 900, and the salient sample set includes a plurality of target image sequences and label information of each target image sequence.

[0159] The first input module 1020 is used to input the target image sequence into the teacher model in the teacher-student network to obtain first sample detection information.

[0160] The second input module 1030 is used to input the target image sequence into the student model in the teacher-student network to obtain second sample detection information.

[0161] The training module 1040 is used to adjust the model parameters of the student model according to the difference between the label information, the first sample detection information and the second sample detection information to obtain a trained deep learning model.

[0162] Fig.11 The block diagram schematically shows a video stream processing device according to an embodiment of the present disclosure.

[0163] like Fig.11 As shown, the video stream processing device 1100 may include a second acquisition module 1110 and a third processing module 1120 .

[0164] The second acquisition module 1110 is used to acquire the video stream to be processed.

[0165] The third processing module 1120 is used to input the video stream to be processed into the trained deep learning model to obtain the video stream processing result.

[0166] The trained deep learning model is obtained by training using the deep learning model training device 1000.

[0167] Fig.12 A block diagram of an electronic device suitable for implementing sample generation, training of a deep learning model, and a video stream processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0168] like Fig.12As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0169] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0170] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as sample generation, deep learning model training, and video stream processing methods. For example, in some embodiments, sample generation, deep learning model training, and video stream processing methods may be implemented as computer software programs, which are tangibly included in machine-readable media, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the sample generation, deep learning model training, and video stream processing methods described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform sample generation, deep learning model training, and video stream processing methods in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0176] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A sample generation method, comprising: Processing M candidate image sequences based on a first method and a second method respectively, obtaining first detection information and second detection information of each candidate image sequence, wherein the first detection information represents whether the candidate image sequence includes at least one of a target object and a motion state of the target object, and the second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and M is a positive integer; Determine N target image sequences from the M candidate image sequences according to the first detection information and the second detection information of each candidate image sequence, where N is a positive integer less than or equal to M; and A salient sample set is generated according to the N target image sequences, the first detection information and the second detection information of each of the target image sequences.

2. The method according to claim 1, wherein: Processing the M candidate image sequences based on the first method includes: Inputting each of the candidate image sequences into a trained large model to obtain the first detection information; Wherein, processing the M candidate image sequences based on the second method includes: Inputting each of the candidate image sequences into a trained deep learning model to obtain initial detection information, wherein the initial detection information represents whether the candidate image sequence includes a target object; and According to the initial detection information, target tracking is performed on a plurality of video frames with consecutive time stamps included in the candidate image sequence to obtain the second detection information; The trained deep learning model is obtained by training using the trained large model.

3. The method according to claim 1, wherein: Determining N target image sequences from the M candidate image sequences according to the first detection information and the second detection information of each candidate image sequence comprises: For each of the candidate image sequences, When the first detection information and the second detection information are consistent, deleting the candidate image sequence; and When the first detection information and the second detection information are inconsistent, the candidate image sequence is determined as the target image sequence.

4. The method according to claim 3, wherein: The first detection information and the second detection information also each include a first confidence level associated with the target object and a second confidence level associated with the motion state; When the first detection information and the second detection information are inconsistent, determining the candidate image sequence as the target image sequence includes: determining a first difference between the first confidence levels and a second difference between the second confidence levels; as well as In response to the first difference value and the second difference value both being greater than or equal to a predetermined confidence threshold, the candidate image sequence is determined as the target image sequence.

5. The method according to any one of claims 1 to 4, wherein: The generating a salient sample set according to the N target image sequences and the first detection information and the second detection information of each target image sequence comprises: for each of the target image sequences, determining label information according to the first detection information and the second detection information, wherein the label information is used to identify whether the target image sequence includes at least one of the target object and the motion state of the target object; and The salient sample set is generated according to each of the target image sequences and respective label information of each of the target image sequences.

6. The method according to any one of claims 1 to 4, further comprising, before processing the M candidate image sequences based on the first manner and the second manner respectively: In response to receiving the video stream to be detected, processing each video frame in the video stream to be detected respectively to obtain a mask image of each video frame; Determine, based on respective binary images of mask images of adjacent video frames, a difference between grayscale values ​​at the same position of the adjacent video frames to obtain a target difference; and In response to the target difference satisfying a predetermined condition, the candidate image sequence is extracted from the video stream to be detected according to the time stamp of the related video frame of the video frame.

7. A method for training a deep learning model, comprising: Acquire a salient sample set, wherein the salient sample set is generated by using the method of claims 1 to 6, and the salient sample set includes a plurality of target image sequences and label information of each of the target image sequences; Inputting the target image sequence into a teacher model in a teacher-student network to obtain first sample detection information; Inputting the target image sequence into the student model in the teacher-student network to obtain second sample detection information; and According to the difference between the label information, the first sample detection information and the second sample detection information, the model parameters of the student model are adjusted to obtain a trained deep learning model.

8. A video stream processing method, comprising: Get the video stream to be processed; as well as Inputting the video stream to be processed into a trained deep learning model to obtain a video stream processing result; Wherein, the trained deep learning model is trained using the method according to claim 7.

9. A sample generating device, comprising: A first processing module is used to process M candidate image sequences based on a first method and a second method respectively, to obtain first detection information and second detection information of each candidate image sequence, wherein the first detection information represents whether the candidate image sequence includes at least one of a target object and a motion state of the target object, and the second detection information represents whether the candidate image sequence includes at least one of the target object and the motion state of the target object, and M is a positive integer; A first determination module is configured to determine N target image sequences from the M candidate image sequences according to the first detection information and the second detection information of each candidate image sequence, where N is a positive integer less than or equal to M; and A generating module is used to generate a significant sample set according to the N target image sequences, the first detection information and the second detection information of each of the target image sequences.

10. The device according to claim 9, wherein: The first processing module comprises: A first input unit, used for inputting each of the candidate image sequences into a trained large model to obtain the first detection information; a second input unit, configured to input each of the candidate image sequences into a trained deep learning model to obtain initial detection information, wherein the initial detection information represents whether the candidate image sequence includes a target object; and a target tracking unit, configured to perform target tracking on a plurality of video frames with consecutive time stamps included in the candidate image sequence according to the initial detection information, to obtain the second detection information; The trained deep learning model is obtained by training using the trained large model.

11. The device according to claim 9, wherein: The first determining module comprises: a deleting unit, configured to, for each of the candidate image sequences, delete the candidate image sequence when the first detection information and the second detection information are consistent; and A first determining unit is configured to determine the candidate image sequence as the target image sequence when the first detection information and the second detection information are inconsistent.

12. The device according to claim 11, wherein The first detection information and the second detection information also each include a first confidence level associated with the target object and a second confidence level associated with the motion state; The first determining unit includes: a first determining subunit, configured to determine a first difference between the first confidence levels and a second difference between the second confidence levels; as well as The second determining subunit is configured to determine the candidate image sequence as the target image sequence in response to the first difference and the second difference being both greater than or equal to a predetermined confidence threshold.

13. The device according to any one of claims 9 to 12, wherein: The generation module comprises: a second determining unit, configured to determine, for each of the target image sequences, label information according to the first detection information and the second detection information, wherein the label information is used to identify whether the target image sequence includes at least one of a target object and a motion state of the target object; and A generating unit is used to generate the salient sample set according to each of the target image sequences and the label information of each of the target image sequences.

14. The device according to any one of claims 9 to 12, further comprising: A second processing module is used for processing each video frame in the video stream to be detected in response to receiving the video stream to be detected, so as to obtain a mask image of each video frame; A second determination module is used to determine the difference between the grayscale values ​​of the adjacent video frames at the same position based on the binary images of the mask images of the adjacent video frames to obtain a target difference; as well as The interception module is used for intercepting the candidate image sequence from the video stream to be detected according to the time stamp of the related video frame of the video frame in response to the target difference satisfying a predetermined condition.

15. A training device for a deep learning model, comprising: a first acquisition module, configured to acquire a salient sample set, wherein the salient sample set is generated by using the apparatus according to claims 9 to 14, and the salient sample set includes a plurality of target image sequences and label information of each of the target image sequences; A first input module, used for inputting the target image sequence into a teacher model in a teacher-student network to obtain first sample detection information; A second input module, used for inputting the target image sequence into the student model in the teacher-student network to obtain second sample detection information; and A training module is used to adjust the model parameters of the student model according to the difference between the label information, the first sample detection information and the second sample detection information to obtain a trained deep learning model.

16. A video stream processing device, comprising: A second acquisition module is used to acquire the video stream to be processed; as well as A third processing module is used to input the video stream to be processed into a trained deep learning model to obtain a video stream processing result; Wherein, the trained deep learning model is trained using the device according to claim 15.

17. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images