Sample generation, model training, action recognition method and device

Abstract

This disclosure provides a sample generation method and apparatus, relating to the field of video processing technology. A specific implementation scheme comprises: obtaining video stream data, annotated with frame intervals for at least one target action; extracting a video frame sequence from the video stream data; generating a confidence vector based on the overlap ratio between the video frame sequence and the frame intervals for each target action in the at least one target action; and annotating the video frame sequence based on the confidence vector to obtain training samples. This approach improves the comprehensiveness and richness of the action information included in the generated training samples.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of video processing technology, and more particularly to a method and device for sample generation, model training, and action recognition. Background Art

[0002] Currently, there are several representative methods for deep learning action recognition, which are mainly used to identify videos that contain a complete action, are short in duration, and have unique labels:

[0003] 1. Spatiotemporal Dual-Stream Approach: In the spatial domain, the appearance of a single frame conveys information about the scene and target depicted in the video. Its static appearance is a useful clue, as certain actions are clearly associated with specific targets. In the temporal domain, the motion of the observer (camera) and the target is expressed in the form of motion across multiple frames.

[0004] 2. Spatial-Temporal Attention Model: The temporal attention component uses an LSTM (Long Short-Term Memory) subnetwork to automatically learn and understand the importance of different frames in the sequence, giving important frames a greater role in classification and optimizing recognition accuracy. For the spatial attention component, an LSTM subnetwork is designed to automatically assign different levels of importance, or attention, to different nodes based on the sequence content.

[0005] 3D Convolutional Network: Replacing the 2D convolution in the classic convolutional neural network with 3D convolution can directly extract the joint features of several consecutive frames, thereby realizing the judgment of actions in the video. Summary of the Invention

[0006] The embodiments of the present disclosure provide a sample generation method, apparatus, device, and storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a sample generation method, which includes: obtaining video stream data, where the video stream data is annotated with a frame number interval of at least one target action; extracting a video frame sequence from the video stream data; generating a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in at least one target action; and annotating the video frame sequence according to the confidence vector to obtain a training sample.

[0008] In the second aspect, an embodiment of the present disclosure provides a model training method, which includes: obtaining a training sample set, where the training sample is a training sample obtained by the method described in any implementation method of the first aspect above; taking the video frame sequence included in the training samples in the training sample set as input, and taking the confidence vector corresponding to the input video frame sequence as the expected output, training the initial action recognition model, and obtaining a trained action recognition model.

[0009] In a third aspect, an embodiment of the present disclosure provides an action recognition method, which includes: obtaining a video frame sequence to be recognized; inputting the video frame sequence to be recognized into an action recognition model to obtain a confidence vector, wherein the action recognition model is an action recognition model obtained by the method described in any implementation method of the second aspect above.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a sample generation device, which includes: a first acquisition module, configured to acquire video stream data, where the video stream data is annotated with a frame number interval of at least one target action; an extraction sequence module, configured to extract a video frame sequence from the video stream data; a generation vector module, configured to generate a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in at least one target action; and a sample determination module, configured to annotate the video frame sequence according to the confidence vector to obtain a training sample.

[0011] In a fifth aspect, an embodiment of the present disclosure provides a model training device, which includes: a second acquisition module, configured to acquire a training sample set, where the training sample is a training sample obtained by the method described in any implementation of the first aspect above; a training model module, configured to take the video frame sequence included in the training sample in the training sample set as input, and take the confidence vector corresponding to the input video frame sequence as the expected output, to train the initial action recognition model, and obtain a trained action recognition model

[0012] In the sixth aspect, an embodiment of the present disclosure provides an action recognition device, which includes: a third acquisition module, configured to acquire a video frame sequence to be recognized; an action recognition module, configured to input the video frame sequence to be recognized into an action recognition model to obtain a confidence vector, wherein the action recognition model is an action recognition model obtained by the method described in any implementation method of the second aspect above.

[0013] In the seventh aspect, an embodiment of the present disclosure provides an electronic device, which includes one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation method of the first aspect, the second aspect, or the third aspect.

[0014] In an eighth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the implementation modes of the first aspect, the second aspect, or the third aspect.

[0015] In a ninth aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any one of the implementation modes of the first aspect, the second aspect, or the third aspect.

[0016] The sample generation method and apparatus provided by the embodiments of the present disclosure effectively improve the comprehensiveness and richness of the action information included in the generated training samples.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0019] Figure 2 is a flow chart of an embodiment of a sample generation method according to the present disclosure;

[0020] Figure 3 is a schematic diagram of an application scenario of the sample generation method according to the present disclosure;

[0021] Figure 4 is a flow chart of another embodiment of a sample generation method according to the present disclosure;

[0022] Figure 5 is a flowchart of an embodiment of a model training method according to the present disclosure;

[0023] Figure 6 is a flowchart of an embodiment of the motion recognition method according to the present disclosure;

[0024] Figure 7 is a schematic diagram of an embodiment of a sample generating apparatus according to the present disclosure;

[0025] Figure 8 is a schematic diagram of an embodiment of a model training device according to the present disclosure;

[0026] Figure 9 is a schematic diagram of an embodiment of a motion recognition device according to the present disclosure;

[0027] Figure 10It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the sample generation method of the present disclosure may be applied.

[0031] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications, such as video playback applications and communication applications, can be installed on terminal devices 101, 102, 103.

[0033] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens, including but not limited to mobile phones and laptop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, to provide sample generation services), or as a single software or software module. No specific limitations are given here.

[0034] Server 105 can be a server that provides various services, for example, obtaining video stream data, where the video stream data is annotated with a frame number interval of at least one target action; extracting a video frame sequence from the video stream data; generating a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in at least one target action; and annotating the video frame sequence according to the confidence vector to obtain a training sample.

[0035] It should be noted that the server 105 can be either hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, to provide sample generation services), or as a single software or software module. No specific limitations are given here.

[0036] It should be noted that the sample generation method, model training method or action recognition method provided by the embodiments of the present disclosure can be executed by the server 105, or by the terminal devices 101, 102, 103, or by the server 105 and the terminal devices 101, 102, 103 in cooperation with each other. Accordingly, the various parts (such as various units, subunits, modules, submodules) included in the sample generation device, model training device, and action recognition device can all be set in the server 105, or all be set in the terminal devices 101, 102, 103, or can be set in the server 105 and the terminal devices 101, 102, 103 respectively.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0038] Figure 2 A flow chart 200 of an embodiment of a sample generation method is shown. The sample generation method comprises the following steps:

[0039] Step 201: Obtain video stream data.

[0040] In this embodiment, the execution subject (e.g., Figure 1 The server 105 or terminal devices 101, 102, 103) can obtain video stream data from local or remote devices.

[0041] The video stream data is usually a long video stream, including one or more target actions and a rich background (frames without target actions). In addition, the video stream is marked with the frame number interval of at least one target action and the corresponding target action identifier.

[0042] Here, the target action can be any action, such as waving, bending, jumping, etc., and this application does not limit this.

[0043] Step 202: extract a video frame sequence from the video stream data.

[0044] In this embodiment, after acquiring the video stream data, the execution subject may randomly extract a video frame sequence from the video stream data, or may extract a video frame sequence around each target action, which is not limited in this application. Here, the video frame sequence is generally a continuous video frame.

[0045] Among them, the video frame sequence may include a preset number of video frames, and the preset number can be set according to actual needs, for example, 5, 10, etc., and this application does not limit this.

[0046] Specifically, the video stream is pre-marked with at least one target action frame interval and the corresponding target action identifier. For example, the frame interval of the target action K is [6, 10]. The execution entity can extract a video frame sequence at a random position in the video stream, for example, video frame [1, 5], or extract a video frame sequence around the target action, for example, video frame [4, 8].

[0047] Step 203 : Generate a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in the at least one target action.

[0048] In this embodiment, after obtaining the video frame sequence, the execution entity can generate a confidence vector based on the overlap rate of the frame number intervals of the video frame sequence and one or more target actions in at least one target action in the video stream data, and the sum of each element of the confidence vector is 1.

[0049] The length of the video frame sequence and the length of the frame interval of the target action may be the same or different, and this application does not impose any limitation on this.

[0050] Specifically, the overlap rate between the video frame sequence and the frame interval of each target action can be calculated by the following formula:

[0051] s∈[a,a+N-1]

[0052] t i ∈[c,c+N-1]

[0053]

[0054] Where s represents the video frame sequence, N is the length of the video frame sequence, and t i represents the video frame interval of the i-th target action; a and c represent s and t respectively i The starting frame of r iIt represents the overlap rate between the video frame sequence in the training sample and the i-th target action.

[0055] In some optional embodiments, a confidence vector is generated based on the overlap rate of the video frame sequence and the frame number interval of each target action in at least one target action, including: generating a confidence vector based on the overlap rate of the video frame sequence and the frame number interval of each target action in at least two target actions.

[0056] In this implementation, the execution entity may generate a confidence vector according to an overlap ratio between the video frame sequence and the frame number interval of each target action in the at least two target actions.

[0057] Specifically, the video stream includes two target actions, target action A and target action B, with corresponding frame intervals of [5, 9] and [11, 15], respectively. The frame interval of the video frame sequence is [8, 12]. Therefore, the overlap rate between the video frame sequence and the frame interval of target action A is 40%, and the overlap rate between the video frame sequence and the frame interval of target action B is 40%. That is, the video frame sequence includes two incomplete target actions A and B. Furthermore, a confidence vector corresponding to the video frame sequence is generated based on the overlap rate.

[0058] This implementation method generates a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in at least two target actions, and then generates a training sample based on the confidence vector, obtaining a training sample containing at least two incomplete target actions and the frame number ratio they occupy.

[0059] Step 204: Label the video frame sequence according to the confidence vector to obtain training samples.

[0060] In this embodiment, the execution entity may annotate the obtained confidence vectors on the corresponding video frame sequence to obtain a video frame sequence annotated with the confidence vectors, and determine the video frame sequence annotated with the confidence vectors as a training sample. The initial action recognition model is then trained based on the training sample to obtain a trained action recognition model.

[0061] Continue to see Figure 3 , Figure 3The figure is a schematic diagram of an application scenario of the sample generation method according to this embodiment. An execution entity 301 may obtain video stream data 302, which is annotated with a frame interval for at least one target action, for example, target action M, with a frame interval of [5, 12]. From the video stream data, a video frame sequence 303 is extracted, for example, with a frame interval of [1, 8]. Based on the overlap rate (e.g., 50%) between video frame sequence 303 and the frame intervals for each target action in the at least one target action, a confidence vector 304 is generated. Video frame sequence 303 is annotated based on confidence vector 304 to obtain training samples.

[0062] The sample generation method provided by the embodiments of the present disclosure obtains video stream data, where the video stream data is annotated with frame intervals of at least one target action; extracts a video frame sequence from the video stream data; generates a confidence vector based on the overlap rate between the video frame sequence and the frame intervals of each target action in at least one target action; annotates the video frame sequence according to the confidence vector to obtain a training sample, so that the generated training sample includes the proportion of at least one target action in the video frame sequence, effectively improving the comprehensiveness and richness of the action information included in the generated training sample.

[0063] Further references Figure 4 , which shows the Figure 2 The process 400 of an embodiment of a sample generation model in the sample generation method embodiment is shown. In this embodiment, the training process of the sample generation model may include the following steps:

[0064] Step 401: Obtain video stream data.

[0065] In this embodiment, the implementation details and technical effects of step 401 can be found in the description of step 201 and will not be repeated here.

[0066] Step 402: extract a video frame sequence from the video stream data.

[0067] In this embodiment, the implementation details and technical effects of step 402 can be found in the description of step 202 and will not be repeated here.

[0068] Step 403 : Generate a confidence vector based on the overlap ratio between the video frame sequence and the frame interval of each target action in the at least one target action, and the overlap ratio between the video frame sequence and the background frame interval.

[0069] In this embodiment, after acquiring the video frame sequence, the execution entity can generate a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in at least one target action, and the overlap rate between the video frame sequence and the background frame number interval.

[0070] Specifically, for example, a video stream includes a target action, target action A, with a corresponding frame interval of [5, 9] and a frame interval of [8, 12]. Therefore, the overlap rate between the video frame sequence and the frame interval of target action A is 40%, while the overlap rate between the video frame sequence and the frame interval of the background (excluding any target action) is 60%. This means that the video frame sequence includes an incomplete target action A and background. Furthermore, a confidence vector is generated based on this overlap rate.

[0071] For example, a video stream contains a target action, target action A, with a corresponding frame interval of [5, 9] and a frame interval of [10, 14]. Therefore, the overlap rate between the video frame sequence and the frame interval of target action A is 0%, while the overlap rate between the video frame sequence and the frame interval of the background is 100%, indicating that the video frame sequence consists solely of the background. Furthermore, based on this overlap rate, a confidence vector corresponding to the video frame sequence is generated.

[0072] In some optional embodiments, a confidence vector is generated based on the overlap rate of the video frame sequence and the frame number interval of each target action in at least one target action, and the overlap rate of the video frame sequence and the background frame number interval, including: generating a confidence vector based on the overlap rate of the video frame sequence and the frame number interval of each target action in at least two target actions, and the overlap rate of the video frame sequence and the background frame number interval.

[0073] In this implementation, the execution entity may be generated according to the overlap ratio between the video frame sequence and the frame intervals of each target action in at least two target actions and the overlap ratio between the video frame sequence and the background frame interval.

[0074] Specifically, the video stream includes two target actions, target action A and target action B, with corresponding frame intervals of [5, 9] and [11, 15], and the frame interval of the video frame sequence is [8, 12]. Therefore, the overlap rate between the video frame sequence and the frame interval of target action A is 40%, the overlap rate between the video frame sequence and the frame interval of target action B is 40%, and the overlap rate between the video frame sequence and the frame interval of the background is 20%. That is, the video frame sequence of the training sample includes two incomplete target actions A and B and a background that does not include any target action. Based on the above overlap rate, a confidence vector corresponding to the video frame sequence is generated.

[0075] This implementation method generates a confidence vector based on the overlap rate between the video frame sequence and the frame number intervals of each target action in at least two target actions, and the overlap rate between the video frame sequence and the background frame number interval, and then generates a training sample based on the confidence vector, obtaining a training sample containing at least two incomplete target actions, the background and the frame number ratio occupied.

[0076] Step 404: Label the video frame sequence according to the confidence vector to obtain training samples.

[0077] In this embodiment, the implementation details and technical effects of step 404 can be found in the description of step 204 and will not be repeated here.

[0078] The above embodiments of the present disclosure, Figure 2 Compared to the corresponding embodiment, process 400 of the sample generation method in this embodiment embodies generating a confidence vector based on the overlap ratio between the video frame sequence and the frame intervals of each target action in at least one target action, as well as the overlap ratio between the video frame sequence and the background frame intervals. Training samples are then obtained based on the confidence vector and the video frame sequence. This embodiment enables the generated training samples to include the proportion of at least one target action and the background in the video frame sequence, further improving the comprehensiveness and richness of the action information included in the generated training samples.

[0079] Continue to refer Figure 5 , shows a process 500 of an embodiment of a model training method according to the present application. The model training method includes the following steps:

[0080] Step 501: Obtain a training sample set.

[0081] In this embodiment, the execution subject can obtain the training sample set at a local or remote device, wherein the training sample is as follows: Figure 2 The training samples obtained by the method described in the corresponding embodiment will not be described in detail here.

[0082] Step 502 : Taking the video frame sequence included in the training sample set as input and the confidence vector corresponding to the input video frame sequence as the expected output, the initial action recognition model is trained to obtain a trained action recognition model.

[0083] In this embodiment, the execution entity can use a machine learning method to take the video frame sequence included in the training samples in the training sample set as input, and the confidence vector corresponding to the input video frame sequence as the expected output to train the initial action recognition model, that is, continuously adjust the model parameters of the initial action recognition model, and in response to determining that the initial action recognition model meets the predetermined training end conditions, obtain a trained action recognition model.

[0084] Here, the initial action recognition model can be an untrained deep learning model (Deep Neural Network, DNN) or an untrained deep learning model. Each layer of the initial action recognition model can be set with initial parameters, and the parameters can be continuously adjusted during the training process. The initial action recognition model can be various types of untrained or untrained artificial neural networks or a model obtained by combining multiple untrained or untrained artificial neural networks. For example, the initial action recognition model can be an untrained convolutional neural network, an untrained recurrent neural network, or a model obtained by combining an untrained convolutional neural network, an untrained recurrent neural network and an untrained fully connected layer.

[0085] Among them, the conditions for the end of training include but are not limited to at least one of the following: the training time reaches a preset duration, the number of training times reaches a preset number, and the function value of a predetermined loss function obtained based on actual output data and expected output data is less than a preset threshold.

[0086] Specifically, the loss function is calculated as follows:

[0087]

[0088] Among them, n is the dimension of the confidence vector, k i are the elements of the confidence vector, For r i The corresponding eigenvalues.

[0089] The model training method provided by the embodiment of the present disclosure obtains a training sample set; takes the video frame sequence included in the training sample in the training sample set as input, takes the confidence vector corresponding to the input video frame sequence as the expected output, trains the initial action recognition model, and obtains a trained action recognition model, wherein the training sample is as follows: Figure 2 The training samples obtained by the method described in the embodiment are helpful in generating an action recognition model for recognizing a video frame sequence including one or more incomplete target actions.

[0090] Continue to refer Figure 6 , shows a process 600 of an embodiment of a motion recognition method according to the present application. The motion recognition method includes the following steps:

[0091] Step 601: Obtain a video frame sequence to be identified.

[0092] In this embodiment, the execution entity may obtain the video frame sequence to be identified through a wired or wireless connection.

[0093] The wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0094] Step 602: Input the video frame sequence to be recognized into the action recognition model to obtain a confidence vector.

[0095] In this embodiment, after acquiring the video frame sequence, the execution subject can input the video frame sequence into the action recognition model to obtain the confidence vector corresponding to the video frame sequence. Figure 5 The action recognition model obtained by the method described in the corresponding embodiment will not be described in detail here.

[0096] The action recognition method provided by the embodiment of the present disclosure obtains a video frame sequence to be recognized; inputs the video frame sequence to be recognized into an action recognition model to obtain a confidence vector, wherein the action recognition model is as follows: Figure 5 The action recognition model obtained by the method described in the embodiment helps to improve the accuracy of recognizing video frame sequences.

[0097] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a sample generation device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0098] like Figure 7 As shown, the sample generation device 700 of this embodiment includes: a first acquisition module 701 , a sequence extraction module 702 , a vector generation module 703 and a sample determination module 704 .

[0099] The first acquisition module 701 may be configured to acquire video stream data.

[0100] The extraction sequence module 702 may be configured to extract a video frame sequence from the video stream data.

[0101] The vector generation module 703 may be configured to generate a confidence vector based on an overlap ratio between the video frame sequence and the frame number interval of each target action in the at least one target action.

[0102] The sample determination module 704 may be configured to determine a video frame sequence and a confidence vector as a training sample.

[0103] In some optional embodiments of this embodiment, the vector generation module is further configured to generate a confidence vector based on the overlap rate between the video frame sequence and the frame number interval of each target action in at least one target action, and the overlap rate between the video frame sequence and the background frame number interval.

[0104] In some optional embodiments of this embodiment, the vector generation module is further configured to generate a confidence vector based on the overlap rate of the video frame sequence and the frame number interval of each target action in at least two target actions, and the overlap rate of the video frame sequence and the background frame number interval.

[0105] In some optional aspects of this embodiment, the vector generation module is further configured to generate a confidence vector based on an overlap ratio between the video frame sequence and the frame number interval of each target action in the at least two target actions.

[0106] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a model training device. Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 8 As shown, the model training device 800 of this embodiment includes: a second acquisition module 801 and a training model module 802.

[0108] The second acquisition module 801 may be configured to acquire a training sample set.

[0109] The training model module 802 can be configured to take the video frame sequence included in the training sample set as input, and the confidence vector corresponding to the input video frame sequence as the expected output, to train the initial action recognition model and obtain a trained action recognition model.

[0110] Further references Figure 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a motion recognition device. Figure 6 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 9 As shown, the action recognition device 900 of this embodiment includes: a third acquisition module 901 and an action recognition module 902 .

[0112] The third acquisition module 901 may be configured to acquire a road image to be classified.

[0113] The action recognition module 902 may be configured to input a video frame sequence to be recognized into an action recognition model to obtain a confidence vector.

[0114] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] like Figure 10 , which is a block diagram of an electronic device according to a sample generation method according to an embodiment of the present disclosure.

[0117] 1000 is a block diagram of an electronic device according to a sample generation method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 10 As shown, the electronic device includes: one or more processors 1001, a memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 10 A processor 1001 is taken as an example.

[0119] Memory 1002 is a non-transitory computer-readable storage medium provided in the present disclosure. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the sample generation method provided in the present disclosure. The non-transitory computer-readable storage medium of the present disclosure stores computer instructions for causing a computer to perform the sample generation method provided in the present disclosure.

[0120] The memory 1002 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the sample generation method in the embodiment of the present disclosure (for example, the attached Figure 7 The processor 1001 executes the non-transient software programs, instructions, and modules stored in the memory 1002 to execute various functional applications and data processing of the server, thereby implementing the sample generation method in the above method embodiment.

[0121] Memory 1002 may include a program storage area and a data storage area. The program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device for face tracking, etc. In addition, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 1002 may optionally include a memory remotely located relative to processor 1001. These remote memories may be connected to the lane detection electronic device via a network. Examples of the aforementioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The electronic device of the sample generation method may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Figure 10 The bus connection is taken as an example.

[0123] The input device 1003 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for lane line detection, such as input devices such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 1004 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0124] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0128] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0129] The technical solution according to the embodiment of the present disclosure helps to improve the comprehensiveness and richness of the action information included in the generated training samples.

[0130] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0131] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A sample generation method, comprising: Acquire video stream data, wherein the video stream data is marked with frame intervals of at least two target actions, and the video stream data includes the at least two target actions and a background; Extracting a video frame sequence from the video stream data, wherein the video frame sequence is continuous video frames; generating a confidence vector based on an overlap ratio between the video frame sequence and the frame number interval of each target action in the at least two target actions, and an overlap ratio between the video frame sequence and the background frame number interval, wherein the overlap ratio between the video frame sequence and the frame number interval of each target action in the at least two target actions is determined according to a ratio of the number of frames occupied by the video frames of each target action in the at least two target actions included in the video frame sequence, and the sum of the elements of the confidence vector is 1; The video frame sequence is labeled according to the confidence vector to obtain a training sample, where the training sample includes at least two incomplete target actions and the proportion of the number of frames occupied by them.

2. A model training method, comprising: Obtaining a training sample set, wherein the training sample is a training sample obtained by the method according to claim 1; The video frame sequence included in the training sample set is used as input, and the confidence vector corresponding to the input video frame sequence is used as the expected output to train the initial action recognition model to obtain a trained action recognition model.

3. A method for motion recognition, comprising: Obtain the video frame sequence to be identified; The video frame sequence to be recognized is input into an action recognition model to obtain a confidence vector, wherein the action recognition model is an action recognition model obtained by the method according to claim 2.

4. A sample generation device, comprising: A first acquisition module is configured to acquire video stream data, wherein the video stream data is marked with frame intervals of at least two target actions, and the video stream data includes the at least two target actions and a background; An extraction sequence module is configured to extract a video frame sequence from the video stream data, wherein the video frame sequence is a continuous video frame; a vector generation module configured to generate a confidence vector based on an overlap ratio between the video frame sequence and the frame number interval of each target action in the at least two target actions, and an overlap ratio between the video frame sequence and the background frame number interval, wherein the overlap ratio between the video frame sequence and the frame number interval of each target action in the at least two target actions is determined according to a ratio of the number of frames occupied by the video frames of each target action in the at least two target actions included in the video frame sequence, and the sum of the elements of the confidence vector is 1; The sample determination module is configured to mark the video frame sequence according to the confidence vector to obtain a training sample, wherein the training sample includes at least two incomplete target actions and the proportion of the number of frames occupied.

5. A model training device comprising: A second acquisition module is configured to acquire a training sample set, wherein the training sample is a training sample obtained by the method according to claim 1; The training model module is configured to take the video frame sequence included in the training sample set as input, take the confidence vector corresponding to the input video frame sequence as the expected output, train the initial action recognition model, and obtain a trained action recognition model.

6. A motion recognition device comprising: A third acquisition module is configured to acquire a sequence of video frames to be identified; The action recognition module is configured to input the video frame sequence to be recognized into an action recognition model to obtain a confidence vector, wherein the action recognition model is an action recognition model obtained by the method according to claim 2.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 3.

9. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 3.

Images