Data enhancement method and device, computer readable storage medium and terminal equipment
By performing subject segmentation and position information calculation of video frames, the action subject is fused with the scene to be fused, the problem of enhancing action detection data in the prior art is solved, high-quality video data suitable for action detection is generated, and the detection effect is improved.
Patent Information
- Application Number
- CN202411010065.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
The existing image enhancement methods mainly focus on static subjects to image enhancement, which is difficult to apply in the training of motion detection, and lack data enhancement methods suitable for motion detection.
By obtaining video frames, subject segmentation and position information calculations are performed, and the action subjects are fusion combined with the preset scenes to be fused, generating a more natural and smooth video frame, reducing the positional mutation of the action subject, which is suitable for action detection.
It realizes the generation of high-quality video data suitable for action detection, and improves the data availability and effect of action detection.
Smart Images

Figure CN118968243A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and in particular, relates to a data enhancement method, device, computer-readable storage medium and terminal device. Background Art
[0002] As an important computer vision task, motion detection has a wide range of application scenarios. For example, in the application scenario of security monitoring, motion detection can be used to determine events such as intrusion, theft, and fall. In the application scenario of human-computer interaction, motion detection can be used to understand the meaning of human actions. In the application scenario of film and television production, motion detection can be used to achieve special effects generation and motion capture.
[0003] Since it is difficult to obtain motion data, it is often necessary to construct sufficient motion data through image enhancement. However, existing image enhancement methods often only perform image enhancement on static subjects, and the obtained image data is difficult to apply to motion detection training. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a data enhancement method, apparatus, computer-readable storage medium and terminal device to solve the problem of lack of data enhancement methods suitable for action detection in the prior art.
[0005] A first aspect of an embodiment of the present application provides a data enhancement method, which may include:
[0006] Acquire a first video frame; wherein the first video frame is any video frame in the first video;
[0007] Performing subject segmentation on the first video frame to obtain various first action subjects;
[0008] Based on an initial video frame, calculating position information of each first action subject; wherein the initial video frame is the first video frame in the first video, the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame;
[0009] Based on the position information of each of the first action subjects, each of the first action subjects is fused with a preset scene to be fused to obtain a second video frame.
[0010] In a specific implementation of the first aspect, calculating the position information of each of the first action subjects based on the initial video frame includes:
[0011] Matching the first action subject with each action subject in the initial video frame to obtain the matched action subject;
[0012] Based on the position of each of the first action subjects and the position of the corresponding matching action subject, the position information of each of the first action subjects is calculated.
[0013] In a specific implementation of the first aspect, after calculating the position information of each of the first action subjects based on the initial video frame, the method further includes:
[0014] Based on the scene to be fused, determining depth information of each action subject in the initial video frame in the scene to be fused;
[0015] Determine, based on the depth information of each action subject in the initial video frame, the depth information of the corresponding first action subject in the scene to be fused;
[0016] The step of fusing each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame includes:
[0017] Based on the position information of each of the first action subjects and the depth information in the scene to be fused, each of the first action subjects is fused with the scene to be fused to obtain the second video frame.
[0018] In a specific implementation of the first aspect, determining, based on the scene to be fused, depth information of each action subject in the initial video frame in the scene to be fused includes:
[0019] Calculating a target vanishing point in the scene to be fused; wherein the target vanishing point is within a fusion range of the scene to be fused;
[0020] Based on the target vanishing point, depth information of each action subject in the initial video frame in the scene to be fused is determined.
[0021] In a specific implementation of the first aspect, the calculating the target vanishing point in the scene to be fused includes:
[0022] Performing straight line detection on the fusion range to obtain straight lines;
[0023] The target vanishing point is determined based on each of the straight lines.
[0024] In a specific implementation of the first aspect, the step of fusing each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame includes:
[0025] Based on the position information of each of the first action subjects, determining a fusion position corresponding to each of the first action subjects in the scene to be fused;
[0026] Based on the corresponding fusion position, each of the first action subjects is affine transformed to obtain the second video frame.
[0027] In a specific implementation of the first aspect, after fusing each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame, the method further includes:
[0028] An enhancement operation of translation, scaling and / or occlusion is performed on the second video frame to obtain the second video frame after the enhancement operation.
[0029] A second aspect of an embodiment of the present application provides a data enhancement device, which may include:
[0030] A first video frame acquisition module, used to acquire a first video frame; wherein the first video frame is any video frame in the first video;
[0031] A subject segmentation module, used to perform subject segmentation on the first video frame to obtain various first action subjects;
[0032] a position information calculation module, configured to calculate the position information of each first action subject based on an initial video frame; wherein the initial video frame is the first video frame in the first video, the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame;
[0033] The scene fusion module is used to fuse each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame.
[0034] In a specific implementation of the second aspect, the location information calculation module includes:
[0035] A subject matching submodule, used for matching the first action subject with each action subject in the initial video frame to obtain the matching action subject;
[0036] The position information calculation submodule is used to calculate the position information of each first action subject based on the position of each first action subject and the position of the corresponding matching action subject.
[0037] In a specific implementation of the second aspect, the data enhancement device further includes:
[0038] A first depth information determination module, configured to determine, based on the scene to be fused, depth information of each action subject in the initial video frame in the scene to be fused;
[0039] A second depth information determination module, configured to determine the depth information of the corresponding first action subject in the to-be-fused scene based on the depth information of each action subject in the initial video frame;
[0040] Furthermore, the scene fusion module is further used to fuse each of the first action subjects with the scene to be fused based on the position information of each of the first action subjects and the depth information in the scene to be fused to obtain the second video frame.
[0041] In a specific implementation of the second aspect, the depth information first determining module includes:
[0042] A target vanishing point calculation submodule, used to calculate a target vanishing point in the scene to be fused; wherein the target vanishing point is within a fusion range of the scene to be fused;
[0043] The depth information determination submodule is used to determine the depth information of each action subject in the initial video frame in the scene to be fused based on the target vanishing point.
[0044] In a specific implementation of the second aspect, the target vanishing point calculation submodule includes:
[0045] A straight line detection unit, used for performing straight line detection on the fusion range to obtain straight lines;
[0046] The target vanishing point determining unit is configured to determine the target vanishing point based on each of the straight lines.
[0047] In a specific implementation of the second aspect, the scene fusion module includes:
[0048] A fusion position determination submodule, used to determine the corresponding fusion position of each of the first action subjects in the to-be-fused scene based on the position information of each of the first action subjects;
[0049] The affine transformation submodule is used to perform affine transformation on each of the first action subjects based on the corresponding fusion position to obtain the second video frame.
[0050] In a specific implementation of the second aspect, the data enhancement device further includes:
[0051] The enhancement operation module is used to perform enhancement operations of translation, scaling and / or occlusion on the second video frame to obtain the second video frame after the enhancement operation.
[0052] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned data enhancement methods are implemented.
[0053] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned data enhancement methods when executing the computer program.
[0054] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any one of the above-mentioned data enhancement methods.
[0055] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a first video frame; wherein the first video frame is any one of the video frames in the first video; the first video frame is segmented to obtain each first action subject; based on the initial video frame, the position information of each first action subject is calculated; wherein the initial video frame is the first video frame in the first video, the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame; based on the position information of each first action subject, each first action subject is fused with a preset scene to be fused to obtain a second video frame. Through the embodiments of the present application, the position information of the first action subject can be obtained according to the displacement of the first action subject relative to the matching action subject in the initial video frame, and fusion can be performed according to the position information to obtain a more natural and smooth video frame, reduce the position mutation of the action subject in the video frame, so that the video data can be applied to action detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0057] Figure 1 This is a flow chart of an embodiment of a data enhancement method in an embodiment of the present application;
[0058] Figure 2 is a schematic diagram of a first video frame;
[0059] Figure 3 A schematic diagram for marking a first video frame;
[0060] Figure 4 A schematic diagram for determining depth information based on a target vanishing point;
[0061] Figure 5 This is a structural diagram of an embodiment of a data enhancement device in an embodiment of the present application;
[0062] Figure 6 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION
[0063] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0064] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0065] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0066] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0067] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0068] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0069] As an important computer vision task, motion detection has a wide range of application scenarios. For example, in the application scenario of security monitoring, motion detection can be used to determine events such as intrusion, theft, and fall. In the application scenario of human-computer interaction, motion detection can be used to understand the meaning of human actions. In the application scenario of film and television production, motion detection can be used to achieve special effects generation and motion capture.
[0070] Since it is difficult to obtain motion data, it is often necessary to construct sufficient motion data through image enhancement. However, existing image enhancement methods often only perform image enhancement on static subjects, and the obtained image data is difficult to apply to motion detection training.
[0071] In view of this, the embodiments of the present application provide a data enhancement method, apparatus, computer-readable storage medium and terminal device to solve the problem of lack of data enhancement methods suitable for action detection in the prior art.
[0072] The execution subject of the embodiment of the present application is a terminal device, specifically, it can be a common computing device such as a desktop computer, a notebook, a handheld computer, or other computing devices, and the specific type of the terminal device is not specifically limited here.
[0073] See also Figure 1 , an embodiment of a data enhancement method in the embodiment of the present application may include:
[0074] Step S101: Acquire a first video frame.
[0075] In an embodiment of the present application, the video that needs to perform action data enhancement can be used as the first video, and the first video can be split into individual video frames according to the frame rate of the video; then, data enhancement can be performed on each video frame in the first video frame by frame.
[0076] In a specific implementation method of an embodiment of the present application, a video including motion data can be cut into videos of a specific length and used as the first video; wherein the specific length can be set according to actual needs, and the present application does not make specific limitations on this. For example, it can be set to a shorter length such as 5 seconds, 10 seconds, 15 seconds, etc.
[0077] For the sake of ease of description, the data enhancement method of an embodiment of the present application will be described below by taking any video frame in the first video (referred to as the first video frame) as an example.
[0078] In the embodiment of the present application, the range of each first action subject in the first video frame can be marked in the form of a detection frame. Here, the subject performing an action in the video frame can be referred to as an action subject, and the action subject in the first video frame can be referred to as a first action subject. The action subject can specifically be a person performing an action. For example, there are two people playing basketball in the first video frame. Figure 2 As shown, the two people playing basketball are the two first action subjects in the first video frame.
[0079] When marking, the action type corresponding to each first action subject can also be marked. For example, if there is a dancing person in the first video frame, the range of the dancing person can be marked in the form of a detection frame, and the corresponding action type can be marked as dancing, such as Figure 3 shown.
[0080] Step S102: segment the first video frame into subjects to obtain first action subjects.
[0081] In the embodiment of the present application, the first video frame may be subject-segmented, and each first action subject may be deducted from the first video frame, so that the deducted first action subject may be subsequently merged with other scenes.
[0082] Among them, the method of subject segmentation can refer to any common segmentation method in the prior art, and this application does not make specific limitations on this. Preferably, the embodiment of the present application can perform subject segmentation on the first video frame based on the model SAM (SegmentAnything Model) for image segmentation to obtain each first action subject; the SAM model can better deduct each first action subject from the first video frame and save it as a foreground image that does not include the background. In another embodiment, the first video frame can be segmented based on other image segmentation models (for example, Grounded-SAM, SegGPT (Segment Everything in Context)) to obtain each first action subject.
[0083] Step S103: Calculate the position information of each first action subject based on the initial video frame.
[0084] In the embodiment of the present application, for each first action subject, the first action subject can be matched with each action subject in the initial video frame to determine the action subject in the initial video frame that matches the first action subject (referred to as a matching action subject). The initial video frame is the first video frame in the first video.
[0085] For example, the initial video frame includes two action subjects, namely action subject 1 and action subject 2, and the first video frame includes two first action subjects, namely action subject 3 and action subject 4; here, action subject 3 can be matched with action subject 1 and action subject 2 in the initial video frame. If action subject 1 matches action subject 3, action subject 1 can be determined as the matching action subject corresponding to action subject 3; if action subject 2 matches action subject 4, action subject 2 can be determined as the matching action subject corresponding to action subject 4.
[0086] Specifically, the action subject can be matched with reference to any matching method in the prior art, and the present application is not limited to this. For example, the Hungarian algorithm, the nearest neighbor algorithm, the Kalman filter or the particle filter can be used to determine the matching action subject. Preferably, in the embodiment of the present application, the Hungarian algorithm can be used to match the first action subject with each action subject in the initial video frame.
[0087] After determining the matching action subject corresponding to the first action subject, the position information of the first action subject may be calculated based on the matching action subject; wherein the position information of the first action subject is the displacement of the first action subject relative to the matching action subject.
[0088] For example, the initial video frame includes two action subjects, namely action subject 1 and action subject 2, and the first video frame includes two first action subjects, namely action subject 3 and action subject 4, and the matching action subject corresponding to action subject 3 is action subject 2, and the matching action subject corresponding to action subject 4 is action subject 1. Then, the displacement of action subject 3 relative to action subject 2 can be calculated, and the displacement can be determined as the position information of action subject 3; the displacement of action subject 4 relative to action subject 1 can also be calculated, and the displacement can be determined as the position information of action subject 4.
[0089] Each video frame in the first video can be calculated according to the above method to obtain the position information of the corresponding action subject in each video frame, and the position information of the action subject in each video frame can be stored in a preset form according to the chronological order of the video frames in the first video. The storage form can be set according to actual needs, and this application does not impose any limitation on this.
[0090] For example, the position information of the corresponding action subject in each video frame may be stored in the form of a list, an array, or a matrix in the order of the first video frame, the second video frame, the third video frame, ... the Nth video frame.
[0091] Based on this, in the subsequent fusion, it is only necessary to determine the position of each action subject in the initial video frame, and then the corresponding position of the action subject in each subsequent video frame can be efficiently determined through the calculated position information; moreover, it can ensure that the action subject can maintain real and natural movement between different video frames, reduce the probability of the action subject jumping between video frames, and help improve the effect of data enhancement.
[0092] Step S104: Based on the position information of each first action subject, each first action subject is fused with a preset scene to be fused to obtain a second video frame.
[0093] In an embodiment of the present application, a background image to be fused can be determined, and the background image can be determined as a scene to be fused. For example, a background image of a grassland can be used as a scene to be fused, and after fusion, a video frame with a grassland background can be obtained. For another example, an image of a shopping mall can be used as a scene to be fused, and after fusion, a video frame with a shopping mall background can be obtained. For another example, a pre-constructed virtual scene can be used as a scene to be fused, and after fusion, a video frame with a specified virtual scene background can be obtained. The scene to be fused can be determined according to demand and is not limited here.
[0094] In order to ensure the authenticity of the fused second video frame, the fusion range in the first action subject can be determined from the preset scene to be fused, and the first action subject and the scene to be fused can be fused within the fusion range to obtain the second video frame.
[0095] Specifically, when determining the fusion range, areas in the scene to be fused where the possibility of the action subject appearing is low can be excluded, and areas in which the possibility of the action subject appearing is high can be used as the fusion range; for example, there are three areas in the scene to be fused, namely, the sky, the wall, and the ground, and the possibility of the action subject (i.e., a person) appearing on the wall or in the sky is small; on the contrary, the possibility of the action subject appearing on the ground is greater, therefore, the ground in the scene to be fused can be used as the fusion range to ensure that the obtained second video frame is more realistic and reliable.
[0096] In an embodiment of the present application, the position of the first action subject in the scene to be fused can be determined based on the position information of the first action subject; then, the first action subject can be integrated into the scene to be fused through affine transformation to obtain a second video frame.
[0097] Specifically, for a certain action subject in the initial video frame, a position can be arbitrarily selected within the fusion range as the position of the action subject in the scene to be fused. The subsequent video frames can determine the fusion position of the action subject in the scene to be fused based on the position information of the action subject in the corresponding video frame.
[0098] For example, the second video frame in the first video includes action subject 2, and action subject 1 in the initial video frame matches action subject 2; when performing data enhancement on the initial video frame, position 1 can be selected in the fusion range as the fusion position of action subject 1 in the scene to be fused; when performing data enhancement on the second video frame, the fusion position of action subject 2 in the scene to be fused can be determined based on the position information of action subject 2; here, the displacement of action subject 2 relative to action subject 1 is 10 cm to the right, then position 2, which is 10 cm to the right of position 1, can be used as the fusion position of action subject 2 in the scene to be fused.
[0099] In a specific implementation method of an embodiment of the present application, in order to reduce the hard edge effect that exists after the first action subject is pasted into the range to be fused, the second video frame can also be processed using a preset blur algorithm (such as Gaussian blur) to soften the edge of the first action subject and the fusion range to obtain a more natural second video frame.
[0100] In another specific implementation of the embodiment of the present application, one or more of the action subjects may be selected to perform the above fusion process. For example, if the initial video frame includes action subject 1, action subject 2, and action subject 3, then only action subject 1 may be data enhanced, and when data enhancement is performed on other video frames, only the action subjects matching action subject 1 in each video frame may be selected for data enhancement.
[0101] In another specific implementation method of the embodiment of the present application, in order to obtain more diverse action data, the depth information of the action subject in the video frame can also be considered to enrich the obtained action data; specifically, based on the scene to be fused, the depth information of each action subject in the initial video frame in the scene to be fused can be determined; then, based on the depth information of each action subject in the initial video frame, the depth information of the corresponding first action subject in the scene to be fused can be determined; then, based on the position information of each first action subject and the depth information in the scene to be fused, each first action subject can be fused with the scene to be fused to obtain a second video frame.
[0102] Generally speaking, for objects with different depth information, their sizes in the picture may be different. Usually, the farther the object is from the camera device, the smaller its size in the picture will be; the closer the object is to the camera device, the larger its size in the picture will be; when the projection lines of parallel or co-pointed lines (such as horizontal lines, vertical lines, etc.) in reality intersect in the video frame, the intersection point obtained is the vanishing point. For example, the tracks of two parallel railways in reality will converge into a point in the video frame, which is the vanishing point; there may be one or more vanishing points in a video frame; here, a vanishing point (called the target vanishing point) can be arbitrarily selected within the fusion range, and the corresponding position of a specific detection frame in the scene to be fused at different depths can be determined according to the target vanishing point, based on which the fusion position and size of the action subject in the initial video frame in the range to be fused can be determined.
[0103] Specifically, taking an action subject in the initial video frame as an example, see Figure 4 When the action subject needs to be pasted into the fusion range, a vanishing point can be randomly selected from each vanishing point in the fusion range as the target vanishing point, and a detection frame 1 can be arbitrarily generated in the fusion range. After that, the four corner points of the detection frame 1 can be connected to the target vanishing point respectively to obtain four lines, and the detection frame 1 can be moved and resized on the four lines so that the four corner points fall on the corresponding lines to generate the detection frame corresponding to the detection frame 1 under different depth information, for example, the detection frame 2 under the depth information 2, the detection frame 3 under the depth information 3 and the detection frame 4 under the depth information 4 are generated; after that, a detection frame can be arbitrarily selected from each generated detection frame as the specific position where the action subject is pasted, and the depth information of the selected detection frame can be used as the depth information of the action subject. For example, the detection frame 3 can be selected as the position where the action subject is pasted, and the depth information 3 can be used as the depth information of the action subject. When fusion is performed, the action subject can be pasted into the detection frame 3 according to the specific position and size of the detection frame 3.
[0104] When processing a first action subject in the first video frame after the initial video frame, a matching action subject corresponding to the first action subject can be determined in the initial video frame, and the depth information corresponding to the matching action subject when fused can be used as the depth information corresponding to the first action subject when fused, thereby constructing each second video frame with consistent depth information and more natural and realistic.
[0105] Based on this, richer enhancement methods can be provided for action data, improving the usability of data enhancement methods.
[0106] The calculation process of the vanishing point is described below.
[0107] Here, the fusion range can be subjected to straight line detection, and the straight lines existing in the fusion range can be detected to obtain the straight lines; specifically, the fusion range can be subjected to edge detection to obtain the edge points in the fusion range; wherein, the edge detection method can adopt any edge detection method in the prior art, for example, Canny edge detector, Sobel edge detector or Prewitt edge detector. Afterwards, the edge points can be mapped from Cartesian space to Hough space, and each edge point is a curve in Hough space; each intersection point in Hough space (i.e., a point where at least two curves pass through) represents a possible straight line between edge points, and the number of curves passing through each intersection point in Hough space can be calculated (called the number of votes for each intersection point), and the intersection point with a number of votes greater than a preset threshold of the number of votes can be used as the target intersection point; wherein, the specific value of the threshold of the number of votes can be set according to actual needs; after determining each target intersection point, each target intersection point can be remapped to Cartesian space to obtain the corresponding straight line.
[0108] Afterwards, the intersection of any two straight lines can be calculated. Specifically, it can be determined by calculating the intersection of the corresponding two straight line equations; based on this, the various intersection points within the fusion range can be calculated, and these intersection points are the vanishing points. Afterwards, any vanishing point can be selected from the various vanishing points within the fusion range as the target vanishing point.
[0109] After determining the target vanishing point, the above method can be used to determine the depth information and fuse the action subject.
[0110] In an embodiment of the present application, each video frame in the first video can be fused with reference to the above method to obtain a corresponding second video frame, and then the fused second video frames can be combined to obtain a second video, thereby obtaining action data arranged according to a timing rule.
[0111] Optionally, after obtaining each second video frame, each second video frame can be subjected to an enhancement operation such as translation, scaling or occlusion as required to obtain each second video frame after the enhancement operation. Afterwards, the second video is composed according to each second video frame after the enhancement operation, thereby increasing the diversity of the action data combination and improving the usability of the data enhancement method.
[0112] Optionally, after obtaining the second video, the second video can be used to perform corresponding model training according to actual needs. For example, the second video can be used as training data to train an artificial intelligence model for monitoring fights between people in the field of security monitoring.
[0113] In summary, the embodiment of the present application obtains a first video frame; wherein, the first video frame is any one video frame in the first video; the first video frame is segmented to obtain each first action subject; based on the initial video frame, the position information of each first action subject is calculated; wherein, the initial video frame is the first video frame in the first video, and the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame; based on the position information of each first action subject, each first action subject is fused with a preset scene to be fused to obtain a second video frame. Through the embodiment of the present application, the position information of the first action subject can be obtained according to the displacement of the first action subject relative to the matching action subject in the initial video frame, and fusion can be performed according to the position information to obtain a more natural and smooth video frame, reducing the position mutation of the action subject in the video frame, so that the video data can be applied to action detection.
[0114] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0115] Corresponding to a data enhancement method described in the above embodiment, Figure 5 A structural diagram of an embodiment of a data enhancement device provided in an embodiment of the present application is shown.
[0116] In an embodiment of the present application, a data enhancement device may include:
[0117] A first video frame acquisition module 501 is used to acquire a first video frame; wherein the first video frame is any video frame in the first video;
[0118] A subject segmentation module 502 is used to segment the first video frame into subjects to obtain first action subjects;
[0119] A position information calculation module 503 is used to calculate the position information of each first action subject based on an initial video frame; wherein the initial video frame is the first video frame in the first video, the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame;
[0120] The scene fusion module 504 is used to fuse each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame.
[0121] In a specific implementation of the embodiment of the present application, the location information calculation module includes:
[0122] A subject matching submodule, used for matching the first action subject with each action subject in the initial video frame to obtain the matching action subject;
[0123] The position information calculation submodule is used to calculate the position information of each first action subject based on the position of each first action subject and the position of the corresponding matching action subject.
[0124] In a specific implementation of the embodiment of the present application, the data enhancement device further includes:
[0125] A first depth information determination module, configured to determine, based on the scene to be fused, depth information of each action subject in the initial video frame in the scene to be fused;
[0126] A second depth information determination module, configured to determine the depth information of the corresponding first action subject in the to-be-fused scene based on the depth information of each action subject in the initial video frame;
[0127] Furthermore, the scene fusion module is further used to fuse each of the first action subjects with the scene to be fused based on the position information of each of the first action subjects and the depth information in the scene to be fused to obtain the second video frame.
[0128] In a specific implementation of the embodiment of the present application, the depth information first determining module includes:
[0129] A target vanishing point calculation submodule, used to calculate a target vanishing point in the scene to be fused; wherein the target vanishing point is within a fusion range of the scene to be fused;
[0130] The depth information determination submodule is used to determine the depth information of each action subject in the initial video frame in the scene to be fused based on the target vanishing point.
[0131] In a specific implementation of the embodiment of the present application, the target vanishing point calculation submodule includes:
[0132] A straight line detection unit, used for performing straight line detection on the fusion range to obtain straight lines;
[0133] The target vanishing point determining unit is configured to determine the target vanishing point based on each of the straight lines.
[0134] In a specific implementation of the embodiment of the present application, the scene fusion module includes:
[0135] A fusion position determination submodule, used to determine the corresponding fusion position of each of the first action subjects in the to-be-fused scene based on the position information of each of the first action subjects;
[0136] The affine transformation submodule is used to perform affine transformation on each of the first action subjects based on the corresponding fusion position to obtain the second video frame.
[0137] In a specific implementation of the embodiment of the present application, the data enhancement device further includes:
[0138] The enhancement operation module is used to perform enhancement operations of translation, scaling and / or occlusion on the second video frame to obtain the second video frame after the enhancement operation.
[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0140] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0141] Figure 6 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0142] like Figure 6 As shown, the terminal device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned data enhancement method embodiments are implemented, for example Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of modules 501 to 504 are shown.
[0143] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the terminal device 6.
[0144] The terminal device 6 may be a computing device such as a desktop computer, a notebook, a PDA, a smart phone, or a smart TV. Those skilled in the art will appreciate that Figure 6 It is only an example of the terminal device 6 and does not constitute a limitation of the terminal device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 6 may also include input and output devices, network access devices, buses, etc.
[0145] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 60 may be the nerve center and command center of the terminal device 6. The processor 60 may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0146] The memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 6. Further, the memory 61 may also include both an internal storage unit of the terminal device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 may also be used to temporarily store data that has been output or is to be output.
[0147] The terminal device 6 may also include a communication module, which may provide communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to network devices. The communication module may be one or more devices integrating at least one communication processing module. The communication module may include an antenna, which may have only one array element or an antenna array including multiple array elements. The communication module may receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module may also receive the signal to be sent from the processor, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna.
[0148] The terminal device 6 may further include a power management module, which may receive input from an external power source, a battery and / or a charger to power the processor, the memory, the communication module, and the like.
[0149] The terminal device 6 may further include a display module, which may be used to display information input by a user or information provided to a user. The display module may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel may cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor to determine the type of the touch event, and then the processor provides a corresponding visual output on the display panel according to the type of the touch event.
[0150] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0153] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0156] An embodiment of the present application provides a computer program product. When the computer program product runs on the terminal device, the terminal device can implement the steps in the above-mentioned various method embodiments.
[0157] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.
[0158] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A data enhancement method, characterized in that: include: Acquire a first video frame; wherein the first video frame is any video frame in the first video; Performing subject segmentation on the first video frame to obtain various first action subjects; Based on an initial video frame, calculating position information of each first action subject; wherein the initial video frame is the first video frame in the first video, the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame; Based on the position information of each of the first action subjects, each of the first action subjects is fused with a preset scene to be fused to obtain a second video frame.
2. The data enhancement method according to claim 1, characterized in that: The calculating, based on the initial video frame, the position information of each of the first action subjects comprises: Matching the first action subject with each action subject in the initial video frame to obtain the matched action subject; Based on the position of each of the first action subjects and the position of the corresponding matching action subject, the position information of each of the first action subjects is calculated.
3. The data enhancement method according to claim 1, characterized in that: After calculating the position information of each of the first action subjects based on the initial video frame, the method further includes: Based on the scene to be fused, determining depth information of each action subject in the initial video frame in the scene to be fused; Determine, based on the depth information of each action subject in the initial video frame, the depth information of the corresponding first action subject in the scene to be fused; The step of fusing each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame includes: Based on the position information of each of the first action subjects and the depth information in the scene to be fused, each of the first action subjects is fused with the scene to be fused to obtain the second video frame.
4. The data enhancement method according to claim 3, characterized in that: The determining, based on the scene to be fused, depth information of each action subject in the initial video frame in the scene to be fused includes: Calculating a target vanishing point in the scene to be fused; wherein the target vanishing point is within a fusion range of the scene to be fused; Based on the target vanishing point, depth information of each action subject in the initial video frame in the scene to be fused is determined.
5. The data enhancement method according to claim 4, characterized in that: The calculating the target vanishing point in the scene to be fused includes: Performing straight line detection on the fusion range to obtain straight lines; The target vanishing point is determined based on each of the straight lines.
6. The data enhancement method according to claim 1, characterized in that: The step of fusing each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame includes: Based on the position information of each of the first action subjects, determining a fusion position corresponding to each of the first action subjects in the scene to be fused; Based on the corresponding fusion position, each of the first action subjects is affine transformed to obtain the second video frame.
7. The data enhancement method according to any one of claims 1 to 6, characterized in that: After fusing each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame, the method further includes: An enhancement operation of translation, scaling and / or occlusion is performed on the second video frame to obtain the second video frame after the enhancement operation.
8. A data enhancement device, characterized in that: include: A first video frame acquisition module, used to acquire a first video frame; wherein the first video frame is any video frame in the first video; A subject segmentation module, used to perform subject segmentation on the first video frame to obtain various first action subjects; a position information calculation module, configured to calculate the position information of each first action subject based on an initial video frame; wherein the initial video frame is the first video frame in the first video, the position information of the first action subject is the displacement of the first action subject relative to the matching action subject, and the matching action subject is the action subject that matches the first action subject in the initial video frame; The scene fusion module is used to fuse each of the first action subjects with a preset scene to be fused based on the position information of each of the first action subjects to obtain a second video frame.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data enhancement method according to any one of claims 1 to 7 are implemented.
10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data enhancement method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Depth extraction method of merging motion information and geometric information
CN101640809A
Video data processing method and device for realizing scene rendering, and calculation equipment
CN107547804A
Multi-exposure shooting method and device and computer readable storage medium
CN111866388A
Real-time high-resolution portrait matting method based on deep neural network
CN116797617A
System and method for generating pan shots from videos
WO2018116322A1
