Motion reconstruction method and device, electronic equipment and computer readable storage medium

By combining traditional cameras with high frame rate brightness change events to generate motion interpolation frames, the problem of discontinuous motion caused by the limited frame rate of traditional cameras is solved, and motion reconstruction with more detailed and richer motion trajectory information is achieved, which is suitable for low power devices.

CN119048651APending Publication Date: 2024-11-29TCL COMMUNICATION TECHNOLOGY (NINGBO) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411095618.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Traditional cameras have a limited frame rate, which leads to poor motion continuity during motion capture and reconstruction.

Method used

Scene images are acquired using a traditional camera with a low frame rate, and motion interpolation frames are generated by combining them with brightness change events with a high frame rate. Frame interpolation processing is then performed to generate composite motion frames for motion reconstruction.

Benefits of technology

It improves the coherence of motion reconstruction, reduces the frame rate requirements of the device, and is suitable for low-power devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048651A_ABST
    Figure CN119048651A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an action reconstruction method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a scene image containing a target object through a first frame rate, and recording a brightness change event corresponding to the scene image through a second frame rate, and the first frame rate is smaller than the second frame rate; generating a basic key frame according to the scene image, and generating an action interpolation frame for the basic key frame according to the brightness change event; performing frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame; and performing action reconstruction on the target object based on the comprehensive action frame to obtain an action reconstruction model. According to the technical scheme, the motion trajectory information of the low-frame-rate key frame is supplemented, the problem that the motion of the low-frame-rate key frame is not coherent during motion reconstruction can be solved through more detailed and richer motion trajectory information, and meanwhile the requirement for the frame rate of equipment for collecting scene images is lowered.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision, and specifically to a method, device, electronic device and computer-readable storage medium for action reconstruction. Background Art

[0002] Computer vision is widely used in motion capture and reconstruction. Currently, traditional cameras (such as RGB cameras) are often used to capture scene images, and deep network models are used to extract or estimate posture key points to achieve motion capture and reconstruction.

[0003] However, due to the limited frame rate of traditional cameras, high frame rate motion capture cannot be achieved, resulting in poor motion continuity after motion reconstruction. Summary of the invention

[0004] The embodiments of the present application provide an action reconstruction method, device, electronic device and computer-readable storage medium, which can improve the continuity of the reconstructed action.

[0005] In a first aspect, an embodiment of the present application provides an action reconstruction method, the method comprising:

[0006] Acquiring a scene image containing a target object at a first frame rate, and recording a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate;

[0007] generating a basic key frame according to the scene image, and generating an action interpolation frame for the basic key frame according to the brightness change event;

[0008] Performing frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame;

[0009] The target object is subjected to motion reconstruction based on the comprehensive motion frame to obtain a motion reconstruction model.

[0010] In a second aspect, an embodiment of the present application further provides an action reconstruction device, the device comprising:

[0011] an acquisition module, configured to acquire a scene image containing a target object at a first frame rate, and record a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate;

[0012] A generating module, configured to generate a basic key frame according to the scene image, and generate an action interpolation frame for the basic key frame according to the brightness change event;

[0013] An interpolation module, used for performing frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame;

[0014] A reconstruction module is used to reconstruct the action of the target object based on the comprehensive action frame to obtain a action reconstruction model.

[0015] Optionally, in some embodiments of the present application, a brightness change event corresponds to brightness change position information and event time information, and the generation module includes:

[0016] A first generating unit, configured to generate basic motion trajectory information according to the basic key frame;

[0017] A second generating unit, configured to generate actual motion trajectory information according to the brightness change position information and the event time information;

[0018] a determination unit, configured to determine the motion trajectory information to be compensated based on a comparison between the actual motion trajectory information and the basic motion trajectory information;

[0019] The third generating unit is used to generate a motion interpolation frame according to the motion trajectory information to be compensated.

[0020] Among them, in some embodiments of the present application, the generation module includes:

[0021] An acquisition unit, used for acquiring a surface electromyography signal of the target object;

[0022] The fourth generating unit is used to generate an action interpolation frame for the basic key frame according to the surface electromyography signal and the brightness change event.

[0023] In some embodiments of the present application, the fourth generating unit includes:

[0024] A classification subunit, used for classifying the surface electromyography signal into action categories to obtain a predicted action type, and determining predicted action trajectory information according to the predicted action type;

[0025] An extraction subunit, configured to generate actual motion trajectory information according to event time information and brightness change position information of the brightness change event;

[0026] A fusion subunit, configured to fuse the predicted motion trajectory information with the actual motion trajectory information corresponding to the brightness change event to obtain fused motion trajectory information;

[0027] A generating subunit is used to generate an action interpolation frame for the basic key frame according to the fused action trajectory information.

[0028] In some embodiments of the present application, the generating subunit is specifically used for:

[0029] Predicting frame brightness information between the basic key frames according to the brightness information of the basic key frames and the fused motion trajectory information;

[0030] Predicting frame color information between the basic key frames according to the color information and brightness information of the basic key frames and the frame brightness information;

[0031] The motion interpolation frames between the basic key frames are generated according to the frame brightness information and the frame color information.

[0032] In some embodiments of the present application, the generating subunit is further specifically used for:

[0033] Correcting the fused motion trajectory information according to the predicted motion type to obtain corrected motion trajectory information;

[0034] Generate an action interpolation frame for the basic key frame according to the corrected action trajectory information.

[0035] In some embodiments of the present application, the reconstruction module includes:

[0036] An alignment unit, used for performing pixel alignment processing on the action interpolation frame in the comprehensive action frame and the basic key frame to obtain an aligned action frame;

[0037] an identification unit, configured to identify object region information of the target object from the aligned action frame;

[0038] An extraction unit, used to extract key point feature information from the object area information;

[0039] The reconstruction unit is used to reconstruct the action of the target object according to the key point feature information in each aligned action frame to obtain an action reconstruction model.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and which implements the steps in the above-mentioned action reconstruction method when the computer program is executed by the processor.

[0041] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned action reconstruction method are implemented.

[0042] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0043] In summary, the embodiments of the present application acquire a scene image including a target object through a first frame rate, and record a brightness change event corresponding to the scene image through a second frame rate, wherein the first frame rate is less than the second frame rate, generate a basic key frame according to the scene image, and generate an action interpolation frame for the basic key frame according to the brightness change event, perform frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame, and reconstruct the action of the target object based on the comprehensive action frame to obtain a motion reconstruction model.

[0044] Among them, by recording the brightness change events corresponding to the scene images, and based on the brightness change events, the frame interpolation processing of the low frame rate basic key frames is performed to obtain the comprehensive action frame, so as to supplement the motion trajectory information of the low frame rate key frames, which helps to solve the problem of motion incoherence of the low frame rate key frames during motion reconstruction through more detailed and richer motion trajectory information, and at the same time, reduce the frame rate requirements of the device for capturing the scene images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a scene schematic diagram of the action reconstruction method provided in an embodiment of the present application;

[0047] Figure 2 is a flowchart of the action reconstruction method provided in an embodiment of the present application;

[0048] Figure 3 is another flowchart of the action reconstruction method provided in an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of the structure of the motion reconstruction device provided in an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] The embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for action reconstruction. Specifically, the embodiments of the present application provide an action reconstruction device suitable for an electronic device, the electronic device including a terminal device or a server, the terminal device including a laptop computer, a desktop computer, a mobile phone or an extended reality device and other devices with a shooting function, wherein the extended reality device includes virtual reality (VR, virtual reality), augmented reality (AR, augmented reality) and mixed reality (MR, mixed reality) and other devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc. The server can be directly or indirectly connected via wired or wireless communication.

[0053] For details, please refer to Figure 1 , Figure 1 : is a schematic diagram of a scenario in which a terminal device according to an embodiment of the present application performs the action reconstruction method, wherein the server performs the action reconstruction method and can be understood by referring to the steps of the terminal device performing the action reconstruction method, wherein the specific execution process of the terminal device performing the action reconstruction method is as follows:

[0054] The terminal device 10 acquires a scene image including a target object at a first frame rate, and records a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate, generates a basic key frame according to the scene image, and generates an action interpolation frame for the basic key frame according to the brightness change event, performs frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame, and reconstructs the action of the target object based on the comprehensive action frame to obtain a motion reconstruction model.

[0055] It can be understood that in the embodiment of the present application, the scene image can be obtained by shooting with a traditional camera with a low frame rate, and the brightness change event can be captured by an event camera.

[0056] In summary, the embodiments of the present application record brightness change events corresponding to scene images, and obtain comprehensive action frames based on the brightness change events by frame interpolation processing of low-frame-rate basic key frames, thereby supplementing the motion trajectory information of low-frame-rate key frames. This helps to solve the problem of motion incoherence of low-frame-rate key frames during motion reconstruction through more detailed and richer motion trajectory information, and at the same time, reduces the requirements on the frame rate of the device for capturing scene images.

[0057] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0058] See also Figure 2 , Figure 2 The flowchart is a schematic diagram of the action reconstruction method provided in the embodiment of the present application. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the specific process of the action reconstruction method is as follows:

[0059] 101. Acquire a scene image containing a target object at a first frame rate, and record a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate.

[0060] In the embodiment of the present application, the first frame rate is less than the second frame rate, that is, the scene image is acquired by a low frame rate conventional camera, and the brightness change event is recorded by a high frame rate. For example, the scene image is acquired by an RGB camera at 30fps, and the brightness change event is recorded by an event camera at a high frame rate.

[0061] Among them, since low-power traditional cameras can be used to capture scene images, the frame rate requirements for the camera that captures scene images are reduced, which is suitable for terminal devices with limited performance. For example, for extended reality devices with limited battery capacity, they can capture scene images through a low frame rate (such as 30fps) acquisition method to meet the performance requirements of motion reconstruction.

[0062] Since event cameras have the characteristics of high frame rate and low power consumption, the use of event cameras can not only collect brightness change events with high frame rate, but also control power consumption. Therefore, using event cameras to collect brightness change events can still meet the requirements of low power consumption.

[0063] Correspondingly, the traditional camera and the event camera can be integrated together on a terminal device for motion reconstruction, so that the terminal device can use the traditional camera and the event camera to achieve motion reconstruction of the target object. For example, the traditional camera (also called an ordinary camera) and the event camera are integrated on an extended display device. The extended reality device can use the traditional camera to obtain scene images through low frame rate acquisition, and use the event camera to obtain brightness change events through high frame rate acquisition.

[0064] 102. Generate a basic key frame according to the scene image, and generate an action interpolation frame for the basic key frame according to the brightness change event.

[0065] It can be understood that the basic key frame describes the key action information for action reconstruction, such as the position information of key points, etc. Among them, although the scene image is obtained based on low frame rate acquisition, since it is directly captured by a visual camera, the scene image can still describe the key information of the target object. Therefore, the scene image can be used as the key frame image for action reconstruction (that is, the basic key frame in the embodiment of the present application).

[0066] Among them, since the brightness change event is acquired based on high frame rate, the brightness change event records more detailed action information than the scene image. For example, based on the characteristics of high frame rate and low frame rate, the brightness change event records the action information between basic key frames. Therefore, action interpolation frames for basic key frames can be generated based on the brightness change event, that is, action interpolation frames that can be inserted between basic key frames to supplement action trajectory information are generated.

[0067] For example, for an animation with 100 frames, the 1st, 25th, 50th, 75th and 100th frames can be understood as basic key frames, and the image frames other than the basic key frames in these 100 frames can be understood as motion interpolation frames.

[0068] 103. Perform frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame.

[0069] The frame interpolation processing of the basic key frame by the action interpolation frame obtains a comprehensive action frame with a higher frame number, wherein the comprehensive action frame has more detailed and richer action trajectory information than the basic key frame. The frame interpolation processing can be interpolated based on time information.

[0070] 104. Reconstruct the action of the target object based on the comprehensive action frame to obtain a action reconstruction model.

[0071] Among them, since the comprehensive action frame has more detailed and richer action trajectory information, the action reconstruction based on the comprehensive action frame is smoother than the action reconstruction based on the basic key frame obtained by traditional camera shooting.

[0072] In summary, the embodiments of the present application record brightness change events corresponding to scene images, and obtain comprehensive action frames based on the brightness change events by frame interpolation processing of low-frame-rate basic key frames, thereby supplementing the motion trajectory information of low-frame-rate key frames. This helps to solve the problem of motion incoherence of low-frame-rate key frames during motion reconstruction through more detailed and richer motion trajectory information, and at the same time, reduces the requirements on the frame rate of the device for capturing scene images.

[0073] Among them, the use of event cameras can also obtain brightness change events with richer motion trajectory information based on the characteristics of high frame rate, and the use of event cameras can also be based on the characteristics of low power consumption. While satisfying the need to record richer brightness change events, it can also control the power consumption of the device, which is suitable for the motion reconstruction needs of low-power devices.

[0074] Optionally, in an embodiment of the present application, actual motion trajectory information corresponding to a brightness change event may be generated, and an action interpolation frame for a basic key frame may be generated based on a difference between the actual motion trajectory information and the basic motion trajectory information corresponding to the basic key frame. That is, optionally, in some embodiments of the present application, a brightness change event corresponds to brightness change position information and event time information, and the step of “generating an action interpolation frame for the basic key frame according to the brightness change event” includes:

[0075] Generate basic motion trajectory information according to the basic key frame;

[0076] Generate actual motion trajectory information according to the brightness change position information and the event time information;

[0077] Determining the motion trajectory information to be compensated according to the comparison between the actual motion trajectory information and the basic motion trajectory information;

[0078] Generate a motion interpolation frame according to the motion trajectory information to be compensated.

[0079] The basic motion trajectory information is based on the motion trajectory change information of each key point or feature point in the basic key frame, and is the position change vector information of the key point or feature point contained in the basic motion frame. The actual motion trajectory information is the motion trajectory change information of each key point or feature point in the brightness change event, and is the position change vector information of the key point or feature point contained in the brightness change event. The position change vector information includes information such as position, speed, acceleration, time and direction.

[0080] It can be understood that by comparing the actual motion trajectory information with the basic motion trajectory information, the difference in motion trajectory information between the two can be determined, and based on the difference in motion trajectory information (motion trajectory information to be compensated), a motion interpolation frame can be generated for insertion between basic key frames.

[0081] For example, for the action trajectory information to be supplemented, based on the basic key frame, the basic key frame is continuously adjusted according to the difference between the action trajectory information to be supplemented and the action trajectory of the basic key frame, and the action interpolation frame is gradually obtained. For example, the position offset of the key point is determined according to the difference between the action trajectory information to be supplemented and the action trajectory of the basic key frame, and the position of the key point in the basic key frame is adjusted based on the position, so as to generate the action interpolation frame based on the basic key frame.

[0082] Optionally, since the brightness change event is an event generated when the brightness in the scene changes, for example, when the light source in the scene flickers, the event camera records the flickering brightness change event, and therefore, the brightness change event can be used to estimate the brightness information of the action interpolation frame between the basic key frames, and then estimate the color information of the action interpolation frame between the basic key frames based on the brightness information and the color information of the basic key frames, and then generate the action interpolation frame based on the brightness information and the color information. That is, optionally, in some embodiments of the present application, the step of "generating the action interpolation frame for the basic key frame according to the brightness change event" includes:

[0083] Determining a target brightness change event located between the basic key frames from the brightness change events according to the timestamp information of the basic key frames;

[0084] Generate segment motion trajectory information according to event time information and brightness change position information of the target brightness change event;

[0085] Predicting frame brightness information at the interpolation moment between the basic key frames according to the brightness information of the basic key frames and the segment motion trajectory information;

[0086] Predicting the frame color information at the insertion moment according to the brightness information and color information of the basic key frame and the frame brightness information;

[0087] The frame brightness information and the frame color information at the interpolation time are synthesized to generate a motion interpolation frame at the interpolation time.

[0088] The segment motion trajectory information describes the motion trajectory change information of the target object between the basic key frames, and is the position change vector information of the key points or feature points included in the target brightness change event.

[0089] Among them, since the brightness of adjacent frames or objects in a short period of time remains basically constant, the frame brightness information of the action interpolation frame adjacent to the basic key frame can be predicted by using the fragment motion trajectory information and the brightness information of the basic key frame. Furthermore, the frame color information of the action interpolation frame can be predicted by using the brightness information and color information of the basic key frame and the frame brightness information of the action interpolation frame.

[0090] In some feasible implementations, for the feature points or key points of the target object, the optical flow vectors of the feature points or key points between the basic key frame and the action interpolation frame are analyzed based on the fragment motion trajectory information, and the frame brightness information and frame color information of the key points or key points in the action interpolation frame are predicted based on the optical flow vectors and the brightness information and color information of the key points or feature points in the basic key frame. The action interpolation frame can be obtained by interpolating and filling the frame brightness information and frame color information of each key point or feature point of the target object.

[0091] It can be understood that the embodiment of the present application uses a high frame rate event camera to capture brightness change events, and generates motion interpolation frames through the brightness change events to supplement the frames of the basic key frames, enriching the details of the motion trajectory and helping to improve the continuity of the reconstructed motion.

[0092] In some other scenes, due to object occlusion or small motion amplitude, it is easy to cause the camera alone to fail to keenly obtain all the details of the motion trajectory changes, resulting in incomplete captured motion trajectory, affecting the accuracy of motion reconstruction. Therefore, in an embodiment of the present application, the surface electromyography signal of the target object is also introduced, and the motion trajectory of the target object is analyzed based on the surface electromyography signal, so as to supplement the problem of incomplete motion trajectory information caused by occlusion or small motion amplitude. That is, optionally, in some embodiments of the present application, the step of "generating an action interpolation frame for the basic key frame according to the brightness change event" includes:

[0093] Acquiring a surface electromyography signal of the target object;

[0094] An action interpolation frame for the basic key frame is generated according to the surface electromyography signal and the brightness change event.

[0095] It can be understood that surface electromyographic signals are the electrophysiological manifestations of muscle fiber action potentials, which reflect the activity level of muscles. Therefore, by analyzing surface electromyographic signals, they can also be used to predict the movement of the target object.

[0096] Among them, since the surface electromyography signal does not rely on visual acquisition technology, there is no situation where it is blocked or the amplitude is small and effective motion data cannot be collected.

[0097] Among them, in an embodiment of the present application, in order to further control the power consumption of action reconstruction, an external surface electromyography signal acquisition device can be used to continuously acquire the surface electromyography signal of the target object, and classify and identify the surface electromyography signal, and only after determining that the action type corresponding to the surface electromyography signal meets the conditions, trigger the traditional camera to acquire the scene image at a low frame rate and trigger the event camera to record the brightness change event at a high frame rate, thereby avoiding the acquisition of invalid scene images and the recording of invalid brightness change events.

[0098] Optionally, in an embodiment of the present application, the surface electromyography signal may be classified and identified to determine the action type corresponding to the surface electromyography signal, and the action interpolation frame may be generated based on the action type. That is, optionally, in some embodiments of the present application, the step of "generating an action interpolation frame for the basic key frame according to the surface electromyography signal and the brightness change event" includes:

[0099] Performing action classification on the surface electromyography signal to obtain a predicted action type, and determining predicted action trajectory information according to the predicted action type;

[0100] Generate actual motion trajectory information according to the event time information and brightness change position information of the brightness change event;

[0101] Fusing the predicted motion trajectory information with the actual motion trajectory information corresponding to the brightness change event to obtain fused motion trajectory information;

[0102] Generate an action interpolation frame for the basic key frame according to the fused action trajectory information.

[0103] Among them, the predicted action type is the action type of the target object predicted based on the surface electromyography signal, for example, the predicted action type includes raising arms, raising hands or running, etc. Among them, since there are differences in the action trajectory information corresponding to each action, the current predicted action trajectory information of the target object can be determined according to the predicted action type. Among them, the predicted action type can be classified and identified by a deep learning model, for example, the deep learning model is trained, the detected surface electromyography signal is input into the deep learning model, and the predicted action type of the surface electromyography signal is output using the deep learning model. Among them, in some embodiments, after predicting the predicted action type of the surface electromyography signal, the motion trajectory of the target object can also be predicted by the deep learning model based on the predicted action type to obtain the predicted action trajectory information. Alternatively, in some embodiments, the surface electromyography signal can also be directly input into the deep learning model, and the characteristics of the surface electromyography signal can be directly analyzed based on the deep learning model, and the predicted action trajectory information corresponding to the surface electromyography signal can be directly output.

[0104] Optionally, in some embodiments, the predicted action trajectory information corresponding to the current predicted action type can be determined by establishing a mapping relationship between the action type and the action trajectory information. For example, for each action type, the corresponding action estimation information is pre-configured. When the predicted action type of the surface electromyography signal is identified, the predicted action trajectory information corresponding to the predicted action type is determined from the mapping relationship.

[0105] It can be understood that the actual motion trajectory information generated based on the brightness change event has the characteristics of being real, high frame rate, continuous and smooth. It is just that in some cases the motion trajectory information is missing due to occlusion, while the predicted motion trajectory information obtained based on the surface electromyography signal prediction has the characteristics of anti-occlusion and continuity. Therefore, the embodiment of the present application realizes compensation of the actual motion trajectory information with the predicted motion trajectory information by fusing the predicted motion trajectory information and the actual motion trajectory information, overcomes the problem of missing motion trajectory information caused by occlusion, and can realize the generation of continuous and smooth motion trajectory information, which in turn helps to achieve the reconstruction of coherent motion after the motion is reconstructed.

[0106] Similarly, in the embodiments of the present application, after the fused motion trajectory information is obtained by fusion, the frame brightness information and frame color information of the motion interpolation frames between the basic key frames can also be estimated based on the brightness information, color information, etc. of the basic key frames, and then the motion interpolation frames between the basic key frames are generated based on the estimated frame brightness information and frame color information. That is, optionally, in some embodiments of the present application, the step of "generating the motion interpolation frames for the basic key frames according to the fused motion trajectory information" includes:

[0107] Predicting frame brightness information between the basic key frames according to the brightness information of the basic key frames and the fused motion trajectory information;

[0108] Predicting frame color information between the basic key frames according to the color information and brightness information of the basic key frames and the frame brightness information;

[0109] The motion interpolation frames between the basic key frames are generated according to the frame brightness information and the frame color information.

[0110] It can be understood that the scheme for generating action interpolation frames based on the fusion motion trajectory information can be understood by referring to the part of "generating action interpolation frames for the basic key frames according to the brightness change event", that is, they are based on the fragment motion trajectory information between the basic key frames, and combined with the color information and brightness information of the basic key frames to estimate the frame brightness information and frame color information of the action difference frame, and then based on the frame brightness information and frame color information, the action interpolation frame is obtained by color space conversion. For example, after obtaining the frame brightness information and frame color information, based on the image conversion relationship between YUV and RGB, the action interpolation frame based on RGB is converted.

[0111] Optionally, in an embodiment of the present application, after the fused motion trajectory information is obtained by fusion, the fused motion trajectory information can also be corrected based on the predicted motion type of the target object to improve the accuracy of the motion trajectory information used for reference to generate the motion interpolation frame. That is, optionally, in some embodiments of the present application, the step of "generating the motion interpolation frame for the basic key frame according to the fused motion trajectory information" includes:

[0112] Correcting the fused motion trajectory information according to the predicted motion type to obtain corrected motion trajectory information;

[0113] Generate an action interpolation frame for the basic key frame according to the corrected action trajectory information.

[0114] It is understandable that by correcting the fused motion trajectory information based on the predicted motion type, the accuracy of the motion trajectory information used to generate the motion interpolation frame can be improved. For example, the unreasonable trajectory information in the fused motion trajectory information can be corrected according to the motion characteristics of the predicted motion type. For example, the first key point with obvious position error in the fused motion trajectory information can be identified according to the motion characteristics, and the position information of the first key point can be adjusted based on the motion characteristics and the position information of the adjacent second key point, so as to achieve the correction effect of the fused motion trajectory information. Alternatively, the smoothness of the fused motion trajectory information can be analyzed according to the motion characteristics corresponding to the predicted motion type, and the incoherent parts can be smoothed to make the transition of the corrected motion trajectory information more natural.

[0115] Correspondingly, after the frame interpolation process obtains the comprehensive action frame, an action reconstruction operation may be performed based on the comprehensive action frame to generate an action reconstruction model for the target object in the device, that is, optionally, in some embodiments of the present application, the step of “performing action reconstruction on the target object based on the comprehensive action frame to obtain an action reconstruction model” includes:

[0116] Performing pixel alignment processing on the action interpolation frame in the comprehensive action frame and the basic key frame to obtain an aligned action frame;

[0117] identifying object region information of the target object from the aligned action frame;

[0118] Extracting key point feature information from the object area information;

[0119] The action of the target object is reconstructed according to the key point feature information in each aligned action frame to obtain an action reconstruction model.

[0120] It can be understood that the object area information corresponds to the foreground area, that is, the area where the target object is located. Since the scene image is shot for the target object and the brightness change event is also for the target object, the object area information can be obtained by extracting the foreground area in the comprehensive action frame, that is, the foreground part of the shot scene image is also mainly the target object.

[0121] Among them, since the action interpolation frame is generated by calculation, in order to improve the accuracy of action reconstruction and improve the smoothness of the reconstructed action, the pixel alignment processing can be performed on each frame image in the comprehensive action frame first, so that the corresponding pixels in multiple frame images can be accurately matched. For example, the foreground area of ​​the target object in each frame image is accurately matched, so as to facilitate the subsequent extraction of key point feature information and action reconstruction for the target object.

[0122] In summary, the embodiments of the present application record brightness change events corresponding to scene images, and obtain comprehensive action frames based on the brightness change events by frame interpolation processing of low-frame-rate basic key frames, thereby supplementing the motion trajectory information of low-frame-rate key frames. This helps to solve the problem of motion incoherence of low-frame-rate key frames during motion reconstruction through more detailed and richer motion trajectory information, and at the same time, reduces the requirements on the frame rate of the device for capturing scene images.

[0123] Among them, the actual motion trajectory information generated based on the brightness change event has the characteristics of authenticity, high frame rate, continuity and smoothness, and the predicted motion trajectory information obtained based on the surface electromyography signal prediction has the characteristics of anti-occlusion and continuity. Therefore, the embodiment of the present application realizes the compensation of the actual motion trajectory information with the predicted motion trajectory information by fusing the predicted motion trajectory information and the actual motion trajectory information, overcomes the problem of missing motion trajectory information caused by occlusion, and can realize the generation of continuous and smooth motion trajectory information, which in turn helps to achieve the reconstruction of coherent motion after the motion is reconstructed.

[0124] Among them, in the embodiment of the present application, the brightness change event can be recorded by a low-power, high-frame-rate event camera, so as to control the power consumption while meeting the requirements of reconstructing the continuity of the action and reduce the power consumption requirements for the device.

[0125] In order to facilitate the understanding of the action reconstruction method of the embodiment of the present application, the action reconstruction method of the embodiment of the present application will be described as a whole below. Figure 3 As shown, Figure 3 is another flow chart of the action reconstruction method provided in an embodiment of the present application, wherein the action reconstruction method includes:

[0126] 201. Collecting surface electromyography signals of a target object by using a surface electromyography signal acquisition device;

[0127] 202, determining whether the action corresponding to the surface electromyography signal is valid, if valid, executing step 203;

[0128] For example, a deep learning model is used to identify the predicted action type of a surface electromyographic signal, and whether the action corresponding to the surface electromyographic signal is valid is determined based on whether a specific action type can be identified. Whether the action is valid means whether it is a valid action, for example, whether it is raising a hand, raising an arm, or raising the head.

[0129] 203. Start a low frame rate traditional camera and a high frame rate event camera;

[0130] 204. Shooting the target object with the conventional camera to obtain a scene image, recording the brightness change event corresponding to the shooting stage of the scene image with the event camera, and analyzing the predicted action type of the surface electromyography signal;

[0131] 205. Filtering scene images as basic key frames, generating actual motion trajectory information according to brightness change events, and generating predicted motion trajectory information according to predicted motion types;

[0132] 206. Fusing the actual motion trajectory information and the predicted motion estimation information to obtain fused motion trajectory information;

[0133] 207. Correcting the fused motion trajectory information according to the predicted motion type to obtain corrected motion trajectory information;

[0134] 208. Estimate frame brightness information of the action interpolation frames between the basic key frames according to the corrected action trajectory information and the brightness information of the basic key frames;

[0135] 209. Estimate frame color information of the action interpolation frame according to the brightness information and color information of the basic key frame and the frame brightness information of the action interpolation frame;

[0136] 210. Obtaining an action interpolation frame by performing image color space conversion according to frame brightness information and frame color information of the action interpolation frame;

[0137] 211. Perform frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame;

[0138] 212. Reconstruct the action of the target object based on the comprehensive action frame to obtain a action reconstruction model.

[0139] Among them, the actual motion trajectory information generated based on the brightness change event has the characteristics of authenticity, high frame rate, continuity and smoothness, and the predicted motion trajectory information obtained based on the surface electromyography signal prediction has the characteristics of anti-occlusion and continuity. Therefore, the embodiment of the present application realizes the compensation of the actual motion trajectory information with the predicted motion trajectory information by fusing the predicted motion trajectory information and the actual motion trajectory information, overcomes the problem of missing motion trajectory information caused by occlusion, and can realize the generation of continuous and smooth motion trajectory information, which in turn helps to achieve the reconstruction of coherent motion after the motion is reconstructed.

[0140] Among them, in the embodiment of the present application, the brightness change event can be recorded by a low-power, high-frame-rate event camera, so as to control the power consumption while meeting the requirements of reconstructing the continuity of the action and reduce the power consumption requirements for the device.

[0141] Among them, the surface electromyography signal acquisition device is an external device of the terminal device that executes the motion reconstruction method of the embodiment of the present application. The surface electromyography signal of the target object is obtained by receiving the surface electromyography signal sent by the external device. Correspondingly, in the embodiment of the present application, after the predicted action type of the target object is analyzed by the surface electromyography signal, the terminal device enables the traditional camera and the event camera, which can avoid the traditional camera and the event camera being in an invalid state of acquisition all the time, effectively control the power consumption of the terminal device, and make the motion reconstruction method suitable for devices with limited power consumption (such as small battery capacity) and performance such as head-mounted devices (such as AR glasses, XR helmets).

[0142] In order to better implement the motion reconstruction method of the present application, the present application also provides a motion reconstruction device based on the above motion reconstruction method. The meanings of the terms are the same as those in the above motion reconstruction method, and the specific implementation details can refer to the description in the method embodiment.

[0143] See also Figure 4 , Figure 4 : is a schematic diagram of the structure of the action reconstruction device provided in an embodiment of the present application, wherein the action reconstruction device may specifically include:

[0144] An acquisition module 301 is configured to acquire a scene image containing a target object at a first frame rate, and record a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate;

[0145] A generating module 302, configured to generate a basic key frame according to the scene image, and generate an action interpolation frame for the basic key frame according to the brightness change event;

[0146] An interpolation module 303 is used to perform frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame;

[0147] The reconstruction module 304 is used to reconstruct the action of the target object based on the comprehensive action frame to obtain a action reconstruction model.

[0148] Optionally, in some embodiments of the present application, a brightness change event corresponds to brightness change position information and event time information, and the generation module 302 includes:

[0149] A first generating unit, configured to generate basic motion trajectory information according to the basic key frame;

[0150] A second generating unit, configured to generate actual motion trajectory information according to the brightness change position information and the event time information;

[0151] a determination unit, configured to determine the motion trajectory information to be compensated based on a comparison between the actual motion trajectory information and the basic motion trajectory information;

[0152] The third generating unit is used to generate a motion interpolation frame according to the motion trajectory information to be compensated.

[0153] In some embodiments of the present application, the generating module 302 includes:

[0154] An acquisition unit, used for acquiring a surface electromyography signal of the target object;

[0155] The fourth generating unit is used to generate an action interpolation frame for the basic key frame according to the surface electromyography signal and the brightness change event.

[0156] In some embodiments of the present application, the fourth generating unit includes:

[0157] A classification subunit, used for classifying the surface electromyography signal into action categories to obtain a predicted action type, and determining predicted action trajectory information according to the predicted action type;

[0158] An extraction subunit, configured to generate actual motion trajectory information according to event time information and brightness change position information of the brightness change event;

[0159] A fusion subunit, configured to fuse the predicted motion trajectory information with the actual motion trajectory information corresponding to the brightness change event to obtain fused motion trajectory information;

[0160] A generating subunit is used to generate an action interpolation frame for the basic key frame according to the fused action trajectory information.

[0161] In some embodiments of the present application, the generating subunit is specifically used for:

[0162] Predicting frame brightness information between the basic key frames according to the brightness information of the basic key frames and the fused motion trajectory information;

[0163] Predicting frame color information between the basic key frames according to the color information and brightness information of the basic key frames and the frame brightness information;

[0164] The motion interpolation frames between the basic key frames are generated according to the frame brightness information and the frame color information.

[0165] In some embodiments of the present application, the generating subunit is further specifically used for:

[0166] Correcting the fused motion trajectory information according to the predicted motion type to obtain corrected motion trajectory information;

[0167] Generate an action interpolation frame for the basic key frame according to the corrected action trajectory information.

[0168] In some embodiments of the present application, the reconstruction module 304 includes:

[0169] An alignment unit, used for performing pixel alignment processing on the action interpolation frame in the comprehensive action frame and the basic key frame to obtain an aligned action frame;

[0170] an identification unit, configured to identify object region information of the target object from the aligned action frame;

[0171] An extraction unit, used to extract key point feature information from the object area information;

[0172] The reconstruction unit is used to reconstruct the action of the target object according to the key point feature information in each aligned action frame to obtain an action reconstruction model.

[0173] In the embodiment of the present application, the acquisition module 301 first acquires a scene image including a target object at a first frame rate, and records a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is less than the second frame rate. Next, the generation module 302 generates a basic key frame according to the scene image, and generates an action interpolation frame for the basic key frame according to the brightness change event. Subsequently, the interpolation module 303 performs frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame. Then, the reconstruction module 304 reconstructs the action of the target object based on the comprehensive action frame to obtain a motion reconstruction model.

[0174] Among them, the embodiment of the present application records the brightness change events corresponding to the scene image, and obtains the comprehensive action frame based on the frame interpolation processing of the low frame rate basic key frame based on the brightness change event, so as to supplement the motion trajectory information of the low frame rate key frame, and helps to solve the problem of motion incoherence of the low frame rate key frame during motion reconstruction through more detailed and richer motion trajectory information, and at the same time, reduce the frame rate requirements of the device for capturing the scene image.

[0175] In addition, the present application also provides an electronic device, such as Figure 5 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0176] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0177] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0178] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0179] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0180] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0181] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby implementing the steps in any one of the action reconstruction methods provided in the embodiments of the present application.

[0182] The embodiment of the present application obtains a scene image including a target object through a first frame rate, and records a brightness change event corresponding to the scene image through a second frame rate, wherein the first frame rate is less than the second frame rate, generates a basic key frame according to the scene image, and generates an action interpolation frame for the basic key frame according to the brightness change event, performs frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame, and reconstructs the action of the target object based on the comprehensive action frame to obtain a motion reconstruction model.

[0183] Among them, by recording the brightness change events corresponding to the scene images, and based on the brightness change events, the frame interpolation processing of the low frame rate basic key frames is performed to obtain the comprehensive action frame, so as to supplement the motion trajectory information of the low frame rate key frames, which helps to solve the problem of motion incoherence of the low frame rate key frames during motion reconstruction through more detailed and richer motion trajectory information, and at the same time, reduce the frame rate requirements of the device for capturing the scene images.

[0184] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0185] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0186] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any action reconstruction method provided in the present application.

[0187] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0188] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0189] Since the instructions stored in the computer-readable storage medium can execute the steps in any action reconstruction method provided in the present application, the beneficial effects that can be achieved by any action reconstruction method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0190] The above is a detailed introduction to a motion reconstruction method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0191] It should be noted that in the specific implementation of the present application, data related to scene images (at least including scene images containing target objects), brightness change events, surface electromyography signals, etc., when the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A motion reconstruction method, characterized in that: The method comprises: Acquiring a scene image containing a target object at a first frame rate, and recording a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate; generating a basic key frame according to the scene image, and generating an action interpolation frame for the basic key frame according to the brightness change event; Performing frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame; The target object is subjected to motion reconstruction based on the comprehensive motion frame to obtain a motion reconstruction model.

2. The motion reconstruction method according to claim 1, characterized in that: The brightness change event corresponds to brightness change position information and event time information, and the step of generating an action interpolation frame for the basic key frame according to the brightness change event includes: Generate basic motion trajectory information according to the basic key frame; generating actual motion trajectory information according to the brightness change position information and the event time information; Determining the motion trajectory information to be compensated according to the comparison between the actual motion trajectory information and the basic motion trajectory information; Generate a motion interpolation frame according to the motion trajectory information to be compensated.

3. The motion reconstruction method according to claim 1, characterized in that: The step of generating an action interpolation frame for the basic key frame according to the brightness change event includes: Acquiring a surface electromyography signal of the target object; An action interpolation frame for the basic key frame is generated according to the surface electromyography signal and the brightness change event.

4. The motion reconstruction method according to claim 3, characterized in that: The step of generating the action interpolation frame for the basic key frame according to the surface electromyography signal and the brightness change event includes: Performing action classification on the surface electromyography signal to obtain a predicted action type, and determining predicted action trajectory information according to the predicted action type; Generate actual motion trajectory information according to the event time information and brightness change position information of the brightness change event; Fusing the predicted motion trajectory information with the actual motion trajectory information corresponding to the brightness change event to obtain fused motion trajectory information; Generate an action interpolation frame for the basic key frame according to the fused action trajectory information.

5. The motion reconstruction method according to claim 4, characterized in that: The step of generating the action interpolation frame for the basic key frame according to the fused action trajectory information includes: Predicting frame brightness information between the basic key frames according to the brightness information of the basic key frames and the fused motion trajectory information; Predicting frame color information between the basic key frames according to the color information and brightness information of the basic key frames and the frame brightness information; The motion interpolation frames between the basic key frames are generated according to the frame brightness information and the frame color information.

6. The motion reconstruction method according to claim 4, characterized in that: The step of generating the action interpolation frame for the basic key frame according to the fused action trajectory information includes: Correcting the fused motion trajectory information according to the predicted motion type to obtain corrected motion trajectory information; Generate an action interpolation frame for the basic key frame according to the corrected action trajectory information.

7. The motion reconstruction method according to claim 1, characterized in that: The step of reconstructing the action of the target object based on the comprehensive action frame to obtain an action reconstruction model includes: Performing pixel alignment processing on the action interpolation frame in the comprehensive action frame and the basic key frame to obtain an aligned action frame; identifying object region information of the target object from the aligned action frame; Extracting key point feature information from the object area information; The action of the target object is reconstructed according to the key point feature information in each aligned action frame to obtain an action reconstruction model.

8. A motion reconstruction device, characterized in that: The device comprises: an acquisition module, configured to acquire a scene image containing a target object at a first frame rate, and record a brightness change event corresponding to the scene image at a second frame rate, wherein the first frame rate is lower than the second frame rate; A generating module, configured to generate a basic key frame according to the scene image, and generate an action interpolation frame for the basic key frame according to the brightness change event; An interpolation module, used for performing frame interpolation processing on the basic key frame according to the action interpolation frame to obtain a comprehensive action frame; A reconstruction module is used to reconstruct the action of the target object based on the comprehensive action frame to obtain a action reconstruction model.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the motion reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the motion reconstruction method according to any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Image processing method and device, electronic equipment and storage medium

    CN120147936A