Video enhancement method and device, medium and program product

Abstract

The invention provides a video enhancement method and device, a medium and a program product, and the method comprises the steps: generating motion parameters for each video, the motion parameters comprising a basic vibration frequency, a vertical bump amplitude and a noise intensity coefficient; generating a vertical displacement and a horizontal displacement of a selected object in the video frame at the t moment according to the generated motion parameters; constructing a transformation matrix according to the horizontal displacement and the vertical displacement; and performing affine transformation on the video frame at the t moment by using the transformation matrix and adopting a predetermined reflection boundary mode to obtain a video frame after motion enhancement. According to the technical scheme, the enhanced video can be obtained by simulating dynamic interference. Furthermore, the rPPG model training data set obtained by using the technical scheme can improve the generalization ability of the rPPG model to the driving scene, and further improve the heart rate measurement robustness of the non-contact heart rate measurement method using the rPPG model.

Description

Technical Field

[0001] This invention relates to the field of computer vision and video processing technology, and in particular to a video enhancement method, apparatus, medium, and program product. Background Technology

[0002] In recent years, non-contact photoplethysmography (rPPG) based on computer vision has developed rapidly. It extracts heart rate signals by capturing periodic color changes or micro-motion features of facial skin microvessels, avoiding the discomfort and operational limitations of traditional contact sensors, and has become a research hotspot in smart healthcare and remote monitoring. However, the pulse wave component in facial color signals is easily coupled in the time-frequency domain with rigid head movements such as translation or rotation, and motion artifacts caused by non-rigid facial expressions such as blinking or speaking. Traditional signal separation methods are difficult to effectively decouple these artifacts, leading to spectral leakage and heart rate estimation bias. Existing rPPG datasets lack training data covering multi-degree-of-freedom motion modes such as oscillatory composite motion, limiting the generalization ability of rPPG models for driving scenarios.

[0003] Therefore, there is a need for video augmentation data that can provide data for shaky scenes in order to generate corresponding model training data, such as training data for rPPG models in shaky scenes, like vehicle driving scenarios. Summary of the Invention

[0004] Embodiments of the present invention provide a video enhancement method, apparatus, medium, and program product that obtains enhanced video by simulating dynamic interference.

[0005] To achieve the above objectives, on the one hand, a video enhancement method is provided, including:

[0006] S1, generate motion parameters for each video, the motion parameters including: basic vibration frequency, vertical sway amplitude and noise intensity coefficient;

[0007] The fundamental vibration frequency is generated using the following formula:

[0008] f b =Uniform(f1,f2)

[0009] f b The fundamental vibration frequency is generated in Hz. Uniform() is a function used to randomly generate floating-point numbers within the given interval. f1 and f2 are the first and second frequencies preset according to the motion disturbance to be simulated.

[0010] Use the following formula to generate the vertical bump amplitude:

[0011] A v=Integer(A1,A2)

[0012] A v The vertical sway amplitude is generated in pixels. Integer() represents a function used to randomly generate integers within the given range, where A1 and A2 are the first and second vertical sway amplitudes preset according to the movement of a predetermined part of the selected object in the video.

[0013] The noise intensity coefficient is generated using the following formula:

[0014] σ = Uniform(σ1, σ2)

[0015] σ is the generated noise intensity coefficient, and σ1 and σ2 are the first noise intensity and the second noise intensity preset according to the change of light intensity at a predetermined part of the selected object;

[0016] S2, based on the generated motion parameters, use the following formula to generate the vertical displacement of the selected object in the video frame at time t. and horizontal displacement ,in:

[0017]

[0018]

[0019] in, B represents the fundamental vibration generated; fps represents the video frame rate. t ·ε t Denotes the generated random shock, where B t ε represents the amplitude of a random impact generated using a predetermined random impact generation method. t ε is the impact strength control coefficient. t Generated using Uniform(0,1); N(0,σ) 2 () represents high-frequency noise; x1 and x2 are the first and second horizontal displacements preset according to the swaying to be simulated;

[0020] S3, based on the horizontal displacement and the vertical displacement Construct the transformation matrix M t ;in,

[0021]

[0022] S4, using the transformation matrix M tThe video frame at time t is subjected to an affine transformation using a predetermined reflection boundary pattern to obtain a motion-enhanced video frame. Specifically, for pixels (x, y) that have exceeded the boundary after the affine transformation, the predetermined reflection boundary pattern is used to map these pixels (x, y) back to the valid region of the video frame at time t.

[0023] x'=max(0,min(2w-x-1,w-1)),

[0024] y'=max(0,min(2h-y-1,h-1)),

[0025] Where w and h are the width and height of the effective region, respectively; x' and y' are the coordinates of the out-of-bounds pixel (x, y) after processing by the reflection boundary mode.

[0026] Preferably, in the video enhancement method, the predetermined reflection boundary pattern is the OpenCV affine transformation function warpAffine, wherein:

[0027] Frame′ t =warpAffine(Frame t ,M,w,h);

[0028] Frame t Let Frame' be the video frame at time t. t For Frame t The video frame after performing an affine transformation.

[0029] Preferably, in the video enhancement method, the shaking to be simulated is shaking that occurs in a driving scenario, including shaking of the head of a person in the vehicle caused by the vibration, shaking or turning of the vehicle.

[0030] Preferably, the video enhancement method further includes one or more of the following:

[0031] f b =Uniform(1.5,2.5);

[0032] A v =Integer(8, 15);

[0033] σ = Uniform(0.3, 1.2);

[0034] B t ∈{0.5,-5}, and trigger a random shock with a predetermined probability;

[0035]

[0036] Preferably, in the video enhancement method, the selected object's predetermined part is a face.

[0037] On the other hand, a video enhancement apparatus is also provided, including a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the steps of the video enhancement method as described in any of the above.

[0038] In another aspect, a computer-readable storage medium is also provided, wherein at least one program is stored therein, the at least one program being executed by a processor to implement the steps of the video enhancement method as described in any of the above descriptions.

[0039] In another aspect, a computer program product is also provided, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the video enhancement methods described above.

[0040] On the other hand, a training method for the rPPG model is also provided, which uses video augmentation methods as described above to generate the training dataset for the rPPG model.

[0041] On the other hand, a non-contact heart rate measurement method is also provided, which uses the rPPG model training method described above to generate the rPPG model training dataset.

[0042] The above technical solution has the following technical effects:

[0043] The technical solution of this invention generates motion parameters including a basic vibration frequency, vertical sway amplitude, and noise intensity coefficient for each video based on the predetermined motion interference to be simulated. The vertical and horizontal displacements of the video frame are generated according to the generated motion parameters to simulate the vibration or swaying distance of the selected object in the vertical and horizontal directions. A transformation matrix is ​​generated accordingly based on the above vertical and horizontal displacements. Then, an affine transformation is performed on the video frame using a predetermined reflection boundary mode and the above transformation matrix to obtain an enhanced video frame with motion interference. Using the enhanced video frame, the problems of motion realism, data diversity, and image integrity can be solved.

[0044] Using a further embodiment of the present invention, motion simulation can be performed on facial swaying caused by vehicle movement in driving scenarios. For example, it can simulate noise generated by car vibration and facial swaying in driving situations. By using video enhancement with the above-mentioned motion simulation, the integrity of facial signals can be suppressed when the facial area moves out of the camera's field of view or undergoes a large-angle deflection in driving scenarios. Therefore, using the video frames enhanced by the above-mentioned video enhancement to train the rPPG model for non-contact heart rate measurement can improve the rPPG model's generalization ability to driving scenarios, enhance the decoupling of motion and physiological signals during rPPG model training, and thus improve the robustness of heart rate detection in facial motion mode. Attached Figure Description

[0045] Figure 1 This is a schematic flowchart of a video enhancement method according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of a video enhancement device according to an embodiment of the present invention. Detailed Implementation

[0047] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0048] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0049] Example 1:

[0050] Figure 1 This is a schematic flowchart of a video enhancement method according to an embodiment of the present invention. Figure 1 The video enhancement method in this embodiment includes:

[0051] S1 generates motion parameters for each video, including: basic vibration frequency, vertical sway amplitude, and noise intensity coefficient;

[0052] The fundamental vibration frequency is generated using the following formula:

[0053] f b =Uniform(f1,f2)

[0054] f bThe fundamental vibration frequency is generated in Hz. `Uniform()` represents a function used to randomly generate floating-point numbers within a given interval. `f1` and `f2` are the first and second frequencies preset according to the motion disturbance to be simulated. In one specific implementation, `f...` b =Uniform(1.5,2.5) is used to simulate the shaking of a person's face when filming a video from a vehicle such as a truck. The range of the basic vibration frequency is set between (1.5, 2.5) based on the natural frequency of the vehicle body.

[0055] Use the following formula to generate the vertical bump amplitude:

[0056] A v =Integer(A1,A2)

[0057] A v The vertical sway amplitude is expressed in pixels. `Integer()` represents a function used to randomly generate integers within a given interval, where A1 and A2 are the first and second vertical sway amplitudes pre-set based on the movement of a predetermined part of a selected object in the video. In one specific implementation, the predetermined part of the selected object is the face within the head. In another specific implementation, the movement of facial feature points from pre-collected videos taken from vehicles such as trucks is statistically analyzed to obtain the interval of the vertical sway amplitude, i.e., the first vertical sway amplitude is 8 pixels, and the second vertical sway amplitude is 15 pixels, i.e., A1 and A2 are the first and second vertical sway amplitudes, respectively. v =Integer(8, 15);

[0058] The noise intensity coefficient is generated using the following formula:

[0059] σ = Uniform(σ1, σ2)

[0060] σ is the generated noise intensity coefficient, and σ1 and σ2 are the first noise intensity and the second noise intensity preset according to the light intensity change of the selected object at a predetermined part; in a specific implementation, the light intensity change of the face in the video taken from a vehicle such as a truck is statistically analyzed in advance to obtain the noise intensity range of (0.3, 1.2), that is, σ = Uniform(0.3, 1.2).

[0061] S2, based on the generated motion parameters, use the following formula to generate the vertical displacement of the selected object in the video frame at time t. and horizontal displacement ,in:

[0062]

[0063]

[0064] in, B represents the fundamental vibration generated; fps represents the video frame rate. t ·ε t Denotes the generated random shock, where B t ε represents the amplitude of a random impact generated using a predetermined random impact generation method. t ε is the impact strength control coefficient. t Generated using Uniform(0,1); N(0,σ) 2 ) represents high-frequency noise; x1 and x2 are the first and second horizontal displacements preset according to the swaying to be simulated; in a specific implementation, B t ∈{0.5,-5}, triggering a random shock with a 25% probability to control the shock intensity; in one specific implementation, The horizontal displacement is measured in pixels, used to simulate vehicle steering or lateral swaying. In one specific implementation, the vertical displacement is measured in pixels.

[0065] S3, based on the horizontal displacement and vertical displacement Construct the transformation matrix M t ;in,

[0066]

[0067] S4, using transformation matrix M t The process involves performing an affine transformation on the video frame at time t using a predetermined reflection boundary pattern to obtain a motion-enhanced video frame. For pixels (x, y) that have exceeded the boundary after the affine transformation, the process further includes mapping these pixels back to the valid region of the video frame at time t using the predetermined reflection boundary pattern.

[0068] x'=max(0,min(2w-x-1,w-1)),

[0069] y'=max(0,min(2h-y-1,h-1)),

[0070] Where w and h are the width and height of the effective region, respectively; x' and y' are the coordinates of the out-of-bounds pixel (x, y) after processing with the reflection boundary mode. An out-of-bounds pixel refers to a pixel whose coordinates exceed the effective region (i.e., the valid range) of the original image after affine transformation. The effective region is the legal coordinate range of the original image, and its range is predetermined, i.e., width 0 ≤ x < w, height 0 ≤ y < h. x' and y' refer to the coordinates of the out-of-bounds pixel after processing with the reflection boundary mode. Through reflection boundary processing, blank or invalid pixels at the image edges after affine transformation can be avoided, and the integrity of the facial region under motion interference can be maintained.

[0071] In one specific implementation, the predetermined reflection boundary pattern is the OpenCV affine transformation function warpAffine, where:

[0072] Frame′ t =warpAffine(Frame t ,M,w,h);

[0073] Frame t Let Frame' be a video frame at time t. t For Frame t The video frame after performing an affine transformation; wherein, this mode uses a local pixel-based resampling interpolation method.

[0074] In this embodiment of the invention, the motion enhancement and edge processing of the video frames were performed using the above steps S3 and S4.

[0075] The embodiments of the present invention achieve motion simulation by adding physical constraints, such as simulating the head movement of a person in a vehicle in a driving scenario.

[0076] This invention generates composite motion trajectories from captured video to simulate noise generated by car vibrations and facial swaying during driving, thus mimicking dynamic interference effects. Specifically, by combining translations in different directions, such as vertical and horizontal, the complex motions of car vibrations or facial swaying are simulated, creating a dynamic interference effect and generating composite motion trajectories from the video. In one specific implementation, multiple transformation matrices are superimposed to generate a simulated motion path and obtain a composite motion trajectory.

[0077] This invention actively introduces motion noise, such as oscillations and / or shaking, through affine transformation, enabling subsequent algorithms that require the enhanced video, such as heart rate detection algorithms, to adapt to such interference during training or inference. This benefits subsequent tasks, such as improving the accuracy of non-contact heart rate detection in driving scenarios.

[0078] Example 2:

[0079] The present invention also provides a video enhancement device, which includes a processor 201, a memory 202, a bus 203, and a computer program stored in the memory 202 and executable on the processor 201. The processor 201 includes one or more processing cores. The memory 202 is connected to the processor 201 via the bus 203. The memory 202 is used to store program instructions. When the processor executes the computer program, it implements the steps in the above-described method embodiments of the present invention.

[0080] Furthermore, as an executable solution, the device can be a computer unit, which may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described structure of the computer unit is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0081] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0082] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0083] Example 3:

[0084] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0085] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0086] Example 4:

[0087] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps described above.

[0088] Example 5:

[0089] The present invention also provides a training method for an rPPG model, which uses any of the video augmentation methods described above to generate a training dataset for the rPPG model.

[0090] The training method of this invention enhances the decoupling of motion-physiological signals in rPPG model training by incorporating motion simulations with physical constraints, such as human head movement simulations in driving scenarios. This overcomes the shortcomings of existing rPPG datasets, which lack training data covering multi-degree-of-freedom motion modes such as oscillatory composite motions, thus limiting the generalization ability of rPPG models to driving scenarios.

[0091] Example 6:

[0092] The present invention also provides a non-contact heart rate measurement method, which uses the rPPG model training method described above to generate a training dataset for the rPPG model.

[0093] The non-contact heart rate measurement method of this invention, by using video enhancement with motion simulation as described above, can suppress the disruption of facial signal integrity when the facial area moves out of the camera's field of view or undergoes a large-angle deflection in driving scenarios. Therefore, using the video frames enhanced by the above video enhancement to train the rPPG model for non-contact heart rate measurement can improve the rPPG model's generalization ability to driving scenarios, enhance the decoupling of motion-physiological signals during rPPG model training, and further improve the robustness of heart rate detection under facial motion modes.

[0094] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A video enhancement method, characterized in that, include: S1, generate motion parameters for each video, the motion parameters including: basic vibration frequency, vertical sway amplitude and noise intensity coefficient; The fundamental vibration frequency is generated using the following formula: f b =Uniform(f1,f2) f b The fundamental vibration frequency is generated in Hz. Uniform() is a function used to randomly generate floating-point numbers within the given interval. f1 and f2 are the first and second frequencies preset according to the motion disturbance to be simulated. Use the following formula to generate the vertical bump amplitude: A v =Integer(A1,A2) A v The vertical sway amplitude is generated in pixels. Integer() represents a function used to randomly generate integers within the given range, where A1 and A2 are the first vertical sway amplitude and the second vertical sway amplitude preset according to the movement of a predetermined part of the selected object in the video. The noise intensity coefficient is generated using the following formula: σ = Uniform(σ1, σ2) σ is the generated noise intensity coefficient, and σ1 and σ2 are the first noise intensity and the second noise intensity preset according to the change of light intensity at a predetermined part of the selected object; S2, based on the generated motion parameters, use the following formula to generate the vertical displacement of the selected object in the video frame at time t. and horizontal displacement in: in, B represents the fundamental vibration generated; fps represents the video frame rate. t ·ε t Denotes the generated random shock, where B t ε represents the amplitude of a random impact generated using a predetermined random impact generation method. t ε is the impact strength control coefficient. t Generated using Uniform(0,1); N(0,σ) 2 () represents high-frequency noise; x1 and x2 are the first and second horizontal displacements preset according to the swaying to be simulated; S3, based on the horizontal displacement and the vertical displacement Construct the transformation matrix M t ;in, S4, using the transformation matrix M t The video frame at time t is subjected to an affine transformation using a predetermined reflection boundary pattern to obtain a motion-enhanced video frame. Specifically, for pixels (x, y) that have exceeded the boundary after the affine transformation, the predetermined reflection boundary pattern is used to map these pixels (x, y) back to the valid region of the video frame at time t. x'=max(0,min(2w-x-1,w-1)), y'=max(0,min(2h-y-1,h-1)), Where w and h are the width and height of the effective region, respectively; x' and y' are the coordinates of the out-of-bounds pixel (x, y) after processing by the reflection boundary mode.

2. The video enhancement method according to claim 1, characterized in that, The predetermined reflection boundary pattern is the OpenCV affine transformation function warpAffine, where: Frame′ t =warpAffine(Frame t ,M,w,h); Frame t Let Frame' be the video frame at time t. t For Frame t The video frame after performing an affine transformation.

3. The video enhancement method according to claim 1, characterized in that, The swaying to be simulated is the swaying that occurs in a driving scenario, including: the swaying of the head of a person in the vehicle caused by the vibration, swaying or turning of the vehicle.

4. The video enhancement method according to claim 3, characterized in that, It also includes one or more of the following: f b =Uniform(1.5,2.5); A v =Integer(8,15); σ = Uniform(0.3, 1.2); B t ∈{0.5,-5}, and trigger a random shock with a predetermined probability; 5. The video enhancement method according to claim 1, characterized in that, The selected object's predetermined part is the face.

6. A video enhancement apparatus, comprising a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the steps of the video enhancement method as claimed in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is executed by a processor to implement the steps of the video enhancement method as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video enhancement method as described in any one of claims 1 to 5.

9. A training method for an rPPG model, characterized in that, The training dataset for the rPPG model is generated using the video enhancement method described in any one of claims 1 to 5.

10. A non-contact heart rate measurement method, characterized in that, The training dataset for the rPPG model is generated using the training method for the rPPG model as described in claim 9.

Images