Video special effects processing method, device, electronic equipment and program product

Abstract

The video special effects processing method, device, electronic device, and program product provided by the embodiments of the present disclosure perform target recognition processing based on at least two types of processing on the driving video collected during the vehicle's driving process, and target tracking processing on the target recognition processing results. In order to render the driving video according to the target to be rendered and the corresponding image trajectory obtained in the driving video, the driving video is rendered, thereby realizing the function of displaying the rendered special effects video. Through this processing method, users can watch the special effects videos taken during their own driving trip, thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing, and more particularly to a method, device, electronic device, and program product for processing video special effects. Background Art

[0002] With the improvement of economic level, more and more people like to travel by car, and it becomes possible to take travel photos of the scenery.

[0003] In the existing technology, travel photography is generally achieved based on mobile terminals or camera devices. After the shooting is completed, the video will be uploaded to the server for processing and display, including special effects. However, this processing process has a strong lag and poor user experience. Summary of the Invention

[0004] In response to the above problems, the embodiments of the present disclosure provide a video special effects processing method, device, electronic device and program product, which provide users with a better viewing experience by real-time acquisition and real-time special effects rendering processing and display of scenes during vehicle driving.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for processing video special effects, including:

[0006] During the driving process of the vehicle, the collected driving video is subjected to target recognition processing based on at least two methods, and the target recognition processing result is subjected to target tracking processing to obtain the target to be rendered in the driving video and the corresponding image trajectory;

[0007] The driving video is rendered according to the image trajectory of the target to be rendered in the driving video, and the special effects video obtained by the special effects rendering process is displayed.

[0008] In a second aspect, an embodiment of the present disclosure provides a video special effects processing device, comprising:

[0009] a rendering module configured to perform target recognition processing based on at least two methods on a collected driving video during vehicle travel, and to perform target tracking processing on the target recognition processing results to obtain a target to be rendered in the driving video and a corresponding image trajectory; and to render the driving video according to the image trajectory of the target to be rendered in the driving video;

[0010] The display module is used to display the special effect video obtained by the special effect rendering process.

[0011] In a third aspect, an embodiment of the present disclosure provides a video special effects processing system, including:

[0012] A vehicle-mounted camera, installed in the driving area of ​​the vehicle, for capturing driving video of the vehicle while it is in motion;

[0013] A vehicle-mounted display device is installed in the driving area of ​​the vehicle and / or the riding area of ​​the vehicle, and is used for the video special effects processing method described in any one of the first aspects, performing special effects rendering processing on the driving video captured by the vehicle-mounted shooting device, and displaying the obtained special effects video.

[0014] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor; and

[0015] Memory;

[0016] The memory stores computer-executable instructions;

[0017] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method as described in any one of the first aspects.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method as described in any one of the first aspects is implemented.

[0019] In a sixth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer instructions, which, when executed by a processor, implement any of the methods described in the first aspect.

[0020] The video special effects processing method, device, electronic device, and program product provided by the embodiments of the present disclosure perform target recognition processing based on at least two types of processing on the driving video collected during the vehicle's driving process, and target tracking processing on the target recognition processing results. In order to render the driving video according to the target to be rendered and the corresponding image trajectory obtained in the driving video, the driving video is rendered, thereby realizing the function of displaying the rendered special effects video. Through this processing method, users can watch the special effects videos taken during their own driving trip, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1A schematic diagram of a scene application of video special effects processing in the prior art;

[0023] Figure 2 A schematic diagram of a network architecture on which the present disclosure is based;

[0024] Figure 3 A flowchart of a video special effects processing method provided by an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a first interface of a video special effects processing method provided by an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a second interface of a video special effects processing method provided by an embodiment of the present disclosure;

[0027] Figure 6A A schematic diagram of a process for identifying and tracking video frames according to an embodiment of the present disclosure;

[0028] Figure 6B A schematic diagram of a video special effects processing method provided by an embodiment of the present disclosure;

[0029] Figure 7 A structural block diagram of a video special effects processing device provided by an embodiment of the present disclosure;

[0030] Figure 8 A structural block diagram of a video special effects processing system provided by an embodiment of the present disclosure;

[0031] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0033] With the improvement of economic level, more and more people like to travel by car, and it becomes possible to take travel photos of the scenery.

[0034] In the prior art, travel photography is generally achieved based on mobile terminals or shooting devices. After the shooting is completed, the video will be uploaded to the server for processing and display including special effects.

[0035] For example, Figure 1This is a schematic diagram of a scene application of video special effects processing in the prior art. Figure 1 As shown, in this scenario, a user takes photos of the scenery while the vehicle is traveling by using a handheld mobile device.

[0036] After completing the shooting, the user can upload the driving video in the mobile device to the special effects processing server in the cloud via the network. The special effects processing server will use the special effects processing algorithm to process the driving video and return the processed special effects video to the mobile device for the user to browse.

[0037] But obviously, for Figure 1 For the scenario shown, the existing method for processing driving videos requires multiple operations such as uploading and downloading video data, and the special effects processing process is relatively cumbersome; on the other hand, the existing video shooting and video special effects processing are performed step by step, and the display of video special effects has a strong lag and a poor user experience.

[0038] To address this issue, according to an embodiment of the present disclosure, an on-board display device that can be used to perform special effects rendering processing on driving videos can be set up on the vehicle, so that while the vehicle is driving, the currently captured driving video can be locally rendered in real time with special effects, and the special effects video after special effects rendering processing can be displayed in real time. In this way, on the one hand, the operation process of users browsing special effects videos can be simplified, and on the other hand, the speed of special effects rendering processing for driving videos can be effectively improved, thereby allowing users to see the special effects video corresponding to the current driving video in real time while the vehicle is driving, thereby improving the user experience.

[0039] refer to Figure 2 , Figure 2 A schematic diagram of a network architecture on which the present disclosure is based. Figure 2 The network architecture shown may specifically include a vehicle 1 , a vehicle-mounted camera device 2 , and a vehicle-mounted display device 3 .

[0040] The vehicle-mounted shooting device 2 may specifically be a driving recorder, a video acquisition device, an image acquisition device, or other hardware device that can be used to shoot and capture scenery along the way while the vehicle is traveling.

[0041] The vehicle-mounted display device 3 can specifically be a hardware device with computing and display functions. By connecting to the vehicle-mounted shooting device 2, it can be used to obtain the driving video shot by the vehicle-mounted shooting device 2 in real time and perform special effects rendering processing on it in real time, and use its display function to display the processed special effects video in real time.

[0042] Based on the aforementioned network architecture, firstly, refer to Figure 3 , Figure 3A flowchart of a video special effects processing method provided by an embodiment of the present disclosure.

[0043] The video special effects processing method provided by the embodiment of the present disclosure includes:

[0044] Step 301: During the driving process of the vehicle, the collected driving video is subjected to target recognition processing based on at least two methods, and the target recognition processing result is subjected to target tracking processing to obtain the target to be rendered in the driving video and the corresponding image trajectory;

[0045] Step 302: Render the driving video according to the image trajectory of the target to be rendered in the driving video, and display the special effects video obtained by the special effects rendering process.

[0046] It should be noted that the execution subject of the video special effects processing method provided in this example is a video special effects processing device installed on a vehicle. Generally, the video special effects processing device will be integrated into a video special effects processing system.

[0047] Combine Figure 2 and Figure 3 As shown, the video special effects processing system includes an on-board camera installed in the vehicle's driving area for capturing driving video of the vehicle while in motion. For example, the on-board camera can be a dashcam installed in the cab, used to record the vehicle's driving process from the vehicle's first-person perspective. The video special effects processing system also includes an on-board display device, which integrates the aforementioned video special effects processing device. Specifically, this can be a central control display device installed in the vehicle's front driving area or a controllable display device installed in the vehicle's rear passenger area.

[0048] When the on-board shooting device is started and collects driving video of the vehicle while driving, the driving video can be sent to the on-board display device in real time. At this time, the video special effects processing device in the on-board display device can perform video special effects processing based on the aforementioned steps 301 and 302, and display the processing results in real time on the on-board display device for users to browse.

[0049] Compared with the existing technology, the localization of video special effects processing eliminates the need for users to upload and download operations, thereby simplifying the process of users browsing special effects videos; in addition, the localization of video special effects processing can also effectively improve the special effects rendering processing speed of driving videos, allowing users to see the special effects video corresponding to the current driving video in real time while the vehicle is driving, thereby improving the user experience.

[0050] Optionally, in the embodiment of the present disclosure, relevant video data obtained during driving can also be synchronously uploaded to the cloud for subsequent viewing and use by users.

[0051] Specifically, the video special effects processing device will respond to the first operation triggered by the user and upload the driving video and / or the special effects video to the cloud for storage. Figure 4 This is a schematic diagram of the first interface of a video special effects processing method provided by an embodiment of the present disclosure, such as Figure 4 As shown, while the video special effects processing device is displaying the special effects video rendered with special effects, the user can trigger the first operation by clicking the "Synchronous Upload" button 401. At this time, the video special effects processing device will synchronously upload the currently captured driving video and / or the currently generated special effects video to the cloud server for storage.

[0052] For example, the server can store multiple special effects videos obtained by performing special effects rendering processing on driving videos collected by a video special effects processing device during different driving processes. While storing the video data of each special effects video itself, the generation time of the special effects video, the generation location of the special effects video (vehicle driving trajectory), and related information such as the vehicle model can also be synchronously stored to facilitate subsequent viewing and use by users.

[0053] Through this "cloud-based" storage method of special effects videos, the video special effects processing device does not need to store the driving videos and special effects videos generated during the historical driving process, thereby effectively saving the storage resources of the video special effects processing device and facilitating the lightweight configuration of the video special effects processing system.

[0054] Optionally, in the disclosed embodiment, the special effects video may also be shared with other users to further enhance the user experience.

[0055] Specifically, the video special effects processing device responds to the second operation triggered by the user and sends the special effects video to other users. Figure 5 A schematic diagram of the second interface of a video special effects processing method provided by an embodiment of the present disclosure, such as Figure 5 As shown, while the video special effects processing device displays the special effects video after special effects rendering, the user can click the "Share" button 501 to trigger the second operation. Figure 5 As shown in the right figure, other users will receive sharing information sent by the video special effects processing device, and through the sharing information, the other users can browse the special effects video synchronously.

[0056] Among them, the sharing information can be specifically various types of sharing information. For example, the sharing information can be an address link, or the sharing information can also be a picture QR code. Other users can browse the special effects video by clicking the address link or scanning the picture QR code.

[0057] By sharing special effects videos in this way, users can not only watch the special effects videos themselves while driving, but also invite other users to watch the special effects videos together, giving the special effects video function more social attributes and improving the user experience.

[0058] Taking into account the high timeliness requirements when performing real-time special effects rendering processing on driving videos, and in order to be able to quickly perform video special effects rendering processing on driving videos, on the basis of the above embodiments, this embodiment will also adopt at least two target recognition algorithms including optical flow prediction algorithm and target detection algorithm to realize target recognition of driving videos, so as to ensure that corresponding special effects rendering processing can be performed subsequently.

[0059] In the above-mentioned special effects rendering processing process, the disclosed embodiment will introduce a target recognition processing method that alternates the optical flow prediction algorithm and the target detection algorithm to improve the processing speed, and cooperate with IOU matching and cascade matching to improve the tracking efficiency, thereby meeting the processing requirements for real-time processing of driving videos.

[0060] Specifically, a driving video is generally composed of multiple continuous video frames. The captured driving video is subjected to at least two target recognition processes, and the target recognition results are subjected to target tracking to obtain the target to be rendered in the driving video and the corresponding image trajectory, including:

[0061] Step 4011: Based on the frame number of the current video frame, a target recognition processing algorithm corresponding to the frame number is used to perform target recognition on the current video frame of the driving video to obtain a target recognition frame in the current video frame.

[0062] Figure 6A A schematic diagram of a process for identifying and tracking video frames provided in an embodiment of the present disclosure, combining steps 4011 and Figure 6A , taking a driving video including M continuous frames as an example, unlike the existing target recognition technology, this embodiment will no longer use a single algorithm to process the video frames, but will use an R-based dynamic algorithm to determine the target recognition frame of each video frame.

[0063] Regarding algorithms, the target detection algorithm has higher detection accuracy but lower efficiency. Conversely, the optical flow prediction algorithm has extremely high detection efficiency but lower accuracy. Considering the characteristics of different algorithms, this embodiment will use an R-based interval call method to call different algorithms for different video frames to perform target recognition frame detection and output.

[0064] In the optional method, the algorithm used for target recognition is determined based on the number of video frames:

[0065] Assuming that the current video frame is the mth frame among M consecutive video frames, when m=1 (i.e., the 1st frame), or when m=1+nR (i.e., the 1st+nR frame), the video special effects processing device will call the target detection algorithm to perform target recognition processing on the mth frame of the current video frame to identify the target recognition box in the mth frame; when m≠1 (i.e., the 1st frame), and when m≠1+nR (i.e., the 1st+nR frame), the video special effects processing device will call the optical flow prediction algorithm to perform target recognition processing on the mth frame of the current video frame to identify the target recognition box in the mth frame; wherein n and R are both positive integers, and R is the frame interval coefficient.

[0066] It can be seen that the optical flow prediction algorithm specifically involves performing sparse optical flow prediction on the position of the target recognition frame in the previous video frame in the current video frame to obtain the target recognition frame in the current video frame. In other words, the accuracy of the target recognition frame obtained by the optical flow prediction algorithm depends to a certain extent on the accuracy of the target recognition frame in the previous frame. Based on this, in this embodiment, the target detection algorithm is called every R frames to detect the target recognition frame with high accuracy on the video frame, thereby improving the efficiency of target recognition frame detection while ensuring accuracy.

[0067] Of course, it should be noted that in the process of using at least two algorithms to perform target recognition processing, the number of target recognition frames obtained by each algorithm processing the video frame is uncertain, that is, the number of target recognition frames can be one or more, and the implementation method of the present application does not limit the number of its target recognition frames.

[0068] Step 4012: predict the actual position of the target to be rendered in the current video frame based on the historical image trajectory of the target to be rendered in the driving video, and obtain the predicted position of the target to be rendered in the current video frame.

[0069] Step 4013: perform matching processing on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame, and obtain the actual position of the target to be rendered in the current video frame according to the matching processing result.

[0070] In combination with step 4012 and step 4013, it can be seen that after completing the determination of the target identification frame of the mth frame, the video special effects processing device will match the target identification frame and the predicted position of each target to be rendered in the mth frame to determine the actual position of each target to be rendered in the mth frame.

[0071] In this embodiment, the matching process will be implemented based on two matching algorithms. First, a first matching process based on spatial overlap is performed on each target recognition frame of the current video frame and the predicted position of each target to be rendered in the current video frame, namely, IOU matching. Then, a second matching process based on feature similarity is performed on each target recognition frame of the current video frame and the predicted position of each target to be rendered in the current video frame, namely, cascade matching.

[0072] Specifically, if Figure 6A As shown, the target recognition frame of the mth frame and the predicted position of the target to be rendered in the mth frame are matched by IOU based on spatial overlap; if the matching result is a match, the actual position of the target to be rendered in the mth frame can be determined based on the matching result; if the matching result is not a match, the target recognition frame of the mth frame and the predicted position of the target to be rendered in the mth frame are matched by cascade matching based on feature similarity, and then the actual position of the target to be rendered in the mth frame is determined based on the cascade matching result.

[0073] It can be seen that IOU matching has a small amount of calculation and its processing efficiency is extremely high, while cascade matching has a better matching effect for occlusion problems. By first using IOU matching and then cascade matching, the matching efficiency can be improved while ensuring matching accuracy.

[0074] In addition, it should be noted that the predicted position of the target to be rendered in the mth frame is obtained by predicting the historical image trajectory of the target to be rendered in the driving video. Generally, the number of targets to be rendered is not fixed, and it may be one or more.

[0075] For example, by utilizing a Kalman filter, the predicted position of each target to be rendered in each video frame of a driving video can be predicted, and the actual position of each video frame obtained by calculating the video special effects processing device is used to update the predicted position of each video frame to ensure the reliability of the predicted position.

[0076] For example, when processing the mth frame, the actual position of each target to be rendered in the 1st frame to the m-1th frame has been calculated and known. At this time, the Kalman filter will predict the position of each target to be rendered in the mth frame to the Mth frame based on the actual position of each target to be rendered in the 1st frame to the m-1th frame, and obtain the predicted position of each target to be rendered in the mth frame to the Mth frame. After completing the processing of the mth frame, the actual position of each target to be rendered in the mth frame will be obtained. The Kalman filter will use the actual position of each target to be rendered in the mth frame to update the predicted position of each target to be rendered in the mth frame previously obtained, and re-predict the position of each target to be rendered in the m+1th frame to the Mth frame. Through this iterative prediction method, it is ensured that the predicted position of each target to be rendered each time is timely.

[0077] Based on the above-mentioned embodiments, in an optional embodiment, the frame interval coefficient R may be a dynamically adjusted value, wherein the value of the frame interval coefficient R is adjusted based on the matching results of the target recognition box of the current video frame and the predicted position of the target to be rendered in the current video frame.

[0078] Combine Figure 6A As shown, when the target recognition frames of the m-th video frame and the predicted positions of the targets to be rendered in each video frame fail to match twice, it means that relative to the previous video frame, a new target to be rendered may appear in the m-th video frame, or the target to be rendered disappears from the m-th video frame.

[0079] At this time, on the one hand, in order to improve recognition stability, it is necessary to use a more accurate target detection algorithm to process subsequent video frames to improve the recognition accuracy of the video as soon as possible, that is, the value of R can be lowered, such as setting R = R-1; on the other hand, it is also necessary to update the predicted position of the rendering target in the video frame to improve the matching efficiency of subsequent video frames.

[0080] After completing the aforementioned processing of the current video frame, the video special effects processing device invokes a video special effects rendering algorithm to process the target to be rendered in the driving video, thereby generating a special effects video. Specifically, the video special effects processing device may invoke the corresponding video special effects rendering algorithm from an algorithm library based on the special effects component selected by the user. Furthermore, when processing each target to be rendered, a special effects image may be superimposed onto the actual position of each target image within the image trajectory of the driving video to generate a special effects video.

[0081] The image trajectory of each target image to be rendered in the driving video consists of the actual position of each target in each video frame. For example, the image trajectory is represented as [target x, Pxm], where Pxm represents the actual position of target x in the mth video frame. For example, for target A, its image trajectory can be represented as [target A, PA1, PA2].

[0082] Based on this, during the special effects processing, the special effects image of target x can be superimposed on the actual position Pxm of target x in the mth video frame to generate the special effects frame image of the mth video frame in the driving video, and the special effects frame images of M video frames can be connected in series to obtain a special effects video.

[0083] Figure 6B A schematic diagram of a video special effects processing method provided by an embodiment of the present disclosure is shown as follows: Figure 6B As shown, interface 601 is a display interface for driving videos, wherein the actual position of the vehicle to be rendered is selected in frame 602, and after special effects rendering processing, a display interface for special effects videos as shown in interface 603 is obtained.

[0084] The video special effects processing method provided by the embodiment of the present disclosure performs real-time special effects rendering processing on the collected driving video during the driving process of the vehicle; and displays the special effects video after special effects rendering processing, so that users can watch the special effects video of travel photography while traveling by car, thereby improving the user experience.

[0085] In the second aspect, corresponding to the video special effects processing method of the above embodiment, Figure 7 This is a structural block diagram of a video special effects processing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7 , a video special effects processing device, comprising:

[0086] The rendering module 710 is configured to perform target recognition processing based on at least two methods on the collected driving video during the vehicle's driving process, and to perform target tracking processing on the target recognition processing results to obtain a target to be rendered in the driving video and a corresponding image trajectory; and to render the driving video according to the image trajectory of the target to be rendered in the driving video;

[0087] The display module 720 is used to display the special effect video obtained by the special effect rendering process.

[0088] In an optional embodiment, the driving video is composed of a plurality of continuous video frames;

[0089] The rendering module 710 is specifically used to: perform target recognition on the current video frame of the driving video according to the frame number of the current video frame using a target recognition processing algorithm corresponding to the frame number, and obtain a target recognition frame in the current video frame; predict the actual position of the target to be rendered in the current video frame according to the historical image trajectory of the target to be rendered in the driving video, and obtain a predicted position of the target to be rendered in the current video frame; perform matching processing on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame, and obtain the actual position of the target to be rendered in the current video frame according to the matching processing result.

[0090] In an optional embodiment, the rendering module 710 is specifically used to: when the current video frame is the first frame in the driving video, or when the current video frame is the 1+nR frame in the driving video, call the target detection algorithm to perform target recognition processing on the current video frame to obtain the target recognition box in the current video frame; wherein, n and R are both positive integers, and R is the frame interval coefficient.

[0091] In an optional embodiment, the rendering module 710 is specifically used to call the optical flow prediction algorithm to perform target recognition processing on the current video frame when the current video frame is not the first frame in the driving video, and the current video frame is not the 1+nR frame in the driving video, so as to obtain the target recognition box in the current video frame; wherein, n and R are both positive integers, and R is the frame interval coefficient.

[0092] In an optional embodiment, the rendering module 710 is specifically configured to perform sparse optical flow prediction on the position of the target recognition frame in the previous video frame in the current video frame to obtain the target recognition frame in the current video frame.

[0093] In an optional embodiment, the rendering module 710 is further used to: adjust the value of the frame interval coefficient R based on the matching result of the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame.

[0094] In an optional embodiment, the rendering module 710 is specifically used to perform a first matching process based on spatial overlap on the target recognition frame of the current video frame and the target to be rendered at the predicted position of the current video frame; if the processing result of the first matching process is a match, the processing result of the first matching process is used as the matching processing result; if the processing result of the first matching process is no match, the target recognition frame of the current video frame and the target to be rendered at the predicted position of the current video frame are performed a second matching process based on feature similarity, and the processing result of the second matching process is used as the matching processing result.

[0095] In an optional embodiment, the rendering module 710 is further configured to call a video special effects rendering algorithm to process the target to be rendered in the driving video to obtain a special effects video.

[0096] In an optional embodiment, the video special effects processing device further includes: an interaction module;

[0097] The interactive module is configured to respond to a first operation triggered by a user and upload the driving video and / or the special effects video to the cloud for storage.

[0098] In an optional embodiment, the video special effects processing device further includes: an interaction module;

[0099] The interaction module is used to respond to a second operation triggered by the user and send the special effects video to other users.

[0100] The video special effects processing device provided by the embodiment of the present disclosure performs real-time special effects rendering processing on the collected driving video during the driving of the vehicle; and displays the special effects video after special effects rendering processing, so that users can watch the special effects video of travel photos while traveling by car, thereby improving the user experience.

[0101] In the third aspect, corresponding to the video special effects processing method of the above embodiment, Figure 8 This is a structural block diagram of a video special effects processing system provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 , video special effects processing system, including:

[0102] The vehicle-mounted camera 810 is installed in the driving area of ​​the vehicle and is used to capture driving video of the vehicle during driving;

[0103] The vehicle-mounted display device 820 is installed in the driving area of ​​the vehicle and / or the riding area of ​​the vehicle, and is used for any of the aforementioned video special effects processing methods to perform special effects rendering processing on the driving video captured by the vehicle-mounted shooting device and display the obtained special effects video.

[0104] The electronic device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0105] refer to Figure 9, which shows a schematic structural diagram of an electronic device 900 suitable for implementing an embodiment of the present disclosure. The electronic device 900 may be a terminal device or a media library. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0106] like Figure 9 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0107] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0108] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0109] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0110] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0111] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0112] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or media library. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] This embodiment provides a computer program product including computer instructions. The computer instructions are executed by a processor as described in any of the preceding methods. The implementation principles and technical effects are similar and will not be described in detail in this embodiment.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0116] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] The following are some embodiments of the present disclosure.

[0119] In a first aspect, according to one or more embodiments of the present disclosure, a method for processing video special effects includes:

[0120] During the driving process of the vehicle, the collected driving video is subjected to target recognition processing based on at least two methods, and the target recognition processing result is subjected to target tracking processing to obtain the target to be rendered in the driving video and the corresponding image trajectory;

[0121] The driving video is rendered according to the image trajectory of the target to be rendered in the driving video, and the special effects video obtained by the special effects rendering process is displayed.

[0122] In an optional embodiment, the driving video is composed of a plurality of continuous video frames;

[0123] Performing target recognition processing based on at least two methods on the collected driving video, and performing target tracking processing on the result of the target recognition processing to obtain a target to be rendered in the driving video and a corresponding image trajectory, including:

[0124] According to the frame number of the current video frame, a target recognition processing algorithm corresponding to the frame number is used to perform target recognition on the current video frame of the driving video to obtain a target recognition frame in the current video frame;

[0125] Predicting the actual position of the target to be rendered in the current video frame based on the historical image trajectory of the target to be rendered in the driving video to obtain the predicted position of the target to be rendered in the current video frame;

[0126] Matching processing is performed on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame, and the actual position of the target to be rendered in the current video frame is obtained according to the matching processing result.

[0127] In an optional embodiment, the target recognition processing algorithm corresponding to the frame number of the current video frame is used to perform target recognition on the current video frame of the driving video to obtain the target recognition frame in the current video frame, including:

[0128] When the current video frame is the first frame in the driving video, or the current video frame is the 1+nR frame in the driving video, calling a target detection algorithm to perform target recognition processing on the current video frame to obtain a target recognition frame in the current video frame;

[0129] Wherein, both n and R are positive integers, and R is a frame interval coefficient.

[0130] In an optional embodiment, the target recognition processing algorithm corresponding to the frame number of the current video frame is used to perform target recognition on the current video frame of the driving video to obtain the target recognition frame in the current video frame, including:

[0131] When the current video frame is not the first frame in the driving video, and the current video frame is not the 1+nR frame in the driving video, calling an optical flow prediction algorithm to perform target recognition processing on the current video frame to obtain a target recognition frame in the current video frame;

[0132] Wherein, both n and R are positive integers, and R is a frame interval coefficient.

[0133] In an optional embodiment, calling an optical flow prediction algorithm to perform target recognition processing on the current video frame to obtain a target recognition frame in the current video frame includes:

[0134] Sparse optical flow prediction is performed on the position of the target recognition frame in the previous video frame in the current video frame to obtain the target recognition frame in the current video frame.

[0135] In an optional embodiment, the method further includes:

[0136] The value of the frame interval coefficient R is adjusted based on a matching result of matching the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame.

[0137] In an optional embodiment, the matching process of the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame includes:

[0138] Performing a first matching process based on spatial overlap on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame;

[0139] If the processing result of the first matching process is a match, using the processing result of the first matching process as the matching processing result;

[0140] If the processing result of the first matching processing is no match, a second matching processing based on feature similarity is performed on the target recognition box of the current video frame and the predicted position of the target to be rendered in the current video frame, and the processing result of the second matching processing is used as the matching processing result.

[0141] In an optional embodiment, the rendering processing of the driving video according to the image trajectory of the target to be rendered in the driving video includes:

[0142] A video special effects rendering algorithm is called to process the target to be rendered in the driving video to obtain a special effects video.

[0143] In an optional embodiment, the method further includes:

[0144] In response to a first operation triggered by a user, the driving video and / or the special effects video are uploaded to the cloud for storage.

[0145] In an optional embodiment, the method further includes:

[0146] In response to a second operation triggered by the user, the special effects video is sent to other users.

[0147] In a second aspect, according to one or more embodiments of the present disclosure, a video special effects processing device is provided, comprising:

[0148] a rendering module configured to perform target recognition processing based on at least two methods on a collected driving video during vehicle travel, and to perform target tracking processing on the target recognition processing results to obtain a target to be rendered in the driving video and a corresponding image trajectory; and to render the driving video according to the image trajectory of the target to be rendered in the driving video;

[0149] The display module is used to display the special effect video obtained by the special effect rendering process.

[0150] In an optional embodiment, the driving video is composed of a plurality of continuous video frames;

[0151] The rendering module is specifically used to: perform target recognition on the current video frame of the driving video according to the frame number of the current video frame using a target recognition processing algorithm corresponding to the frame number, and obtain a target recognition frame in the current video frame; predict the actual position of the target to be rendered in the current video frame according to the historical image trajectory of the target to be rendered in the driving video, and obtain a predicted position of the target to be rendered in the current video frame; perform matching processing on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame, and obtain the actual position of the target to be rendered in the current video frame according to the matching processing result.

[0152] In an optional embodiment, the rendering module is specifically used to: when the current video frame is the first frame in the driving video, or when the current video frame is the 1+nR frame in the driving video, call the target detection algorithm to perform target recognition processing on the current video frame to obtain the target recognition box in the current video frame; wherein, n and R are both positive integers, and R is the frame interval coefficient.

[0153] In an optional embodiment, the rendering module is specifically used to call the optical flow prediction algorithm to perform target recognition processing on the current video frame when the current video frame is not the first frame in the driving video, and the current video frame is not the 1+nR frame in the driving video, so as to obtain the target recognition box in the current video frame; wherein, n and R are both positive integers, and R is the frame interval coefficient.

[0154] In an optional embodiment, the rendering module is specifically configured to perform sparse optical flow prediction on the position of the target recognition frame in the previous video frame in the current video frame to obtain the target recognition frame in the current video frame.

[0155] In an optional embodiment, the rendering module is further used to: adjust the value of the frame interval coefficient R based on the matching result of the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame.

[0156] In an optional embodiment, the rendering module is specifically used to perform a first matching process based on spatial overlap on the target recognition frame of the current video frame and the target to be rendered at the predicted position of the current video frame; if the processing result of the first matching process is a match, the processing result of the first matching process is used as the matching processing result; if the processing result of the first matching process is no match, the target recognition frame of the current video frame and the target to be rendered at the predicted position of the current video frame are performed a second matching process based on feature similarity, and the processing result of the second matching process is used as the matching processing result.

[0157] In an optional embodiment, the rendering module is further used to call a video special effects rendering algorithm to process the target to be rendered in the driving video to obtain a special effects video.

[0158] In an optional embodiment, the video special effects processing device further includes: an interaction module;

[0159] The interactive module is configured to respond to a first operation triggered by a user and upload the driving video and / or the special effects video to the cloud for storage.

[0160] In an optional embodiment, the video special effects processing device further includes: an interaction module;

[0161] The interaction module is used to respond to a second operation triggered by the user and send the special effects video to other users.

[0162] In a third aspect, according to one or more embodiments of the present disclosure, a video special effects processing system is provided, comprising:

[0163] A vehicle-mounted camera, installed in the driving area of ​​the vehicle, for capturing driving video of the vehicle while it is in motion;

[0164] A vehicle-mounted display device is installed in the driving area of ​​the vehicle and / or the passenger area of ​​the vehicle, and is used to use the video special effects processing method described in any of the above items to perform special effects rendering processing on the driving video captured by the vehicle-mounted shooting device and display the obtained special effects video.

[0165] In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0166] The memory stores computer-executable instructions;

[0167] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs any of the methods described above.

[0168] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method described in any of the preceding items is implemented.

[0169] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer program product includes computer instructions, and the computer instructions are executed by a processor to perform the method described in any of the preceding items.

[0170] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0171] In addition, although adopting specific order to describe operation, this should not be interpreted as requiring these operations to be carried out in the specific order shown or in sequential order.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the present disclosure.Some features described in the context of separate embodiment can also be implemented in single embodiment in combination.On the contrary, the features described in the context of single embodiment also can be implemented in multiple embodiments individually or in the mode of any suitable subcombination.

[0172] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video special effects processing method, characterized in that: The method is applied to a vehicle-mounted display device, comprising: During the driving process of the vehicle, target recognition processing is performed on the collected driving video based on at least two algorithms, and target tracking processing is performed on the target recognition processing result to obtain the target to be rendered in the driving video and the corresponding image trajectory; According to the image trajectory of the target to be rendered in the driving video, the driving video is rendered, and the special effects video obtained by the special effects rendering is displayed; wherein, The target recognition processing based on at least two algorithms on the collected driving video includes: alternately calling the target detection algorithm and the optical flow prediction algorithm to perform target recognition processing based on the frame interval coefficient; wherein the inter-frame coefficient is dynamically adjusted according to the matching result between the target recognition box of the video frame and the predicted position of the video frame in the driving video.

2. The method according to claim 1, characterized in that The driving video is composed of a plurality of continuous video frames; Performing target recognition processing based on at least two algorithms on the collected driving video, and performing target tracking processing on the result of the target recognition processing to obtain a target to be rendered in the driving video and a corresponding image trajectory, including: According to the frame number of the current video frame, a target recognition processing algorithm corresponding to the frame number is used to perform target recognition on the current video frame of the driving video to obtain a target recognition frame in the current video frame; Predicting the actual position of the target to be rendered in the current video frame based on the historical image trajectory of each target to be rendered in the driving video to obtain the predicted position of the target to be rendered in the current video frame; Matching processing is performed on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame, and the actual position of the target to be rendered in the current video frame is obtained according to the matching processing result.

3. The method according to claim 2, characterized in that The method of performing target recognition on the current video frame of the driving video according to the frame number of the current video frame using a target recognition processing algorithm corresponding to the frame number to obtain a target recognition frame in the current video frame includes: When the current video frame is the first frame in the driving video, or the current video frame is the 1+nR frame in the driving video, calling a target detection algorithm to perform target recognition processing on the current video frame to obtain a target recognition frame in the current video frame; Wherein, both n and R are positive integers, and R is a frame interval coefficient.

4. The method according to claim 2, characterized in that The method of performing target recognition on the current video frame of the driving video according to the frame number of the current video frame using a target recognition processing algorithm corresponding to the frame number to obtain a target recognition frame in the current video frame includes: When the current video frame is not the first frame in the driving video, and the current video frame is not the 1+nR frame in the driving video, calling an optical flow prediction algorithm to perform target recognition processing on the current video frame to obtain a target recognition frame in the current video frame; Wherein, both n and R are positive integers, and R is a frame interval coefficient.

5. The method according to claim 4, characterized in that Calling an optical flow prediction algorithm to perform target recognition processing on the current video frame to obtain a target recognition frame in the current video frame, including: Sparse optical flow prediction is performed on the position of the target recognition frame in the previous video frame in the current video frame to obtain the target recognition frame in the current video frame.

6. The method according to claim 2, characterized in that The matching process of the target recognition frame of the current video frame and the target to be rendered at the predicted position of the current video frame includes: Performing a first matching process based on spatial overlap on the target recognition frame of the current video frame and the predicted position of the target to be rendered in the current video frame; If the processing result of the first matching process is a match, using the processing result of the first matching process as the matching processing result; If the processing result of the first matching processing is no match, a second matching processing based on feature similarity is performed on the target recognition box of the current video frame and the predicted position of the target to be rendered in the current video frame, and the processing result of the second matching processing is used as the matching processing result.

7. The method according to claim 1, characterized in that The rendering process of the driving video according to the image trajectory of the target to be rendered in the driving video includes: A video special effects rendering algorithm is called to process the target to be rendered in the driving video to obtain a special effects video.

8. The method according to any one of claims 1 to 7, characterized in that Also includes: In response to a first operation triggered by a user, the driving video and / or the special effects video are uploaded to the cloud for storage.

9. The method according to any one of claims 1 to 7, characterized in that Also includes: In response to a second operation triggered by the user, the special effects video is sent to other users.

10. A video special effects processing device, characterized in that: The device is provided on a vehicle-mounted display device and includes: a rendering module configured to perform target recognition processing based on at least two methods on a collected driving video during vehicle travel, and to perform target tracking processing on the target recognition processing results to obtain a target to be rendered in the driving video and a corresponding image trajectory; and to render the driving video according to the image trajectory of the target to be rendered in the driving video; A display module is used to display the special effects video obtained by special effects rendering processing; Among them, the rendering module is further used to: alternately call the target detection algorithm and the optical flow prediction algorithm to perform target recognition processing based on the frame interval coefficient; wherein, the inter-frame coefficient is dynamically adjusted according to the matching result of the target recognition box of the video frame and the predicted position of the video frame in the driving video.

11. A video special effects processing system, characterized in that: include: A vehicle-mounted camera, installed in the driving area of ​​the vehicle, for capturing driving video of the vehicle while it is in motion; A vehicle-mounted display device is installed in the driving area of ​​the vehicle and / or the passenger area of ​​the vehicle, and is used to use the video special effects processing method described in any one of claims 1 to 9 to perform special effects rendering processing on the driving video captured by the vehicle-mounted shooting device, and display the obtained special effects video.

12. An electronic device, wherein: include: at least one processor; as well as Memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images