Video synthesis method of camera device, camera device, and storage medium

Abstract

The application discloses a video synthesis method of a camera device, the camera device and a storage medium. The method comprises the following steps: collecting a recording video stream in the process of collecting a monitoring video stream for video monitoring; the video parameter of the recording video stream is lower than the video parameter of the monitoring video stream; extracting a target video frame containing a target object in the recording video stream to obtain a video library for editing; synthesizing at least part of the target video frame into a target video based on the video library; and sending the target video to a user terminal. In the foregoing manner, the camera device can complete the synthesis of the video, the problem of excessive processing pressure and playing lag caused by the user terminal for video synthesis can be reduced, and the probability of playing lag can be reduced.

Description

Technical Field

[0001] This application relates to the field of video compositing technology, and in particular to video compositing methods, camera equipment, and storage media for camera equipment. Background Technology

[0002] Whether in real life or online, creating beautiful videos from various photos and videos is a popular trend. In modern society, video production is increasingly popular, and video editing software is emerging in large numbers. Nowadays, in addition to video editing software that needs to be downloaded to a computer, some mobile video editing software is gradually becoming popular, but most of them still require users to edit and produce the videos themselves.

[0003] Current camera equipment, especially home monitoring and pet cameras, allows users to easily view the situation in the monitored area and also makes it easy for users to create corresponding video clips from the monitored footage. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a video synthesis method, a camera device, and a storage medium for video equipment, which can improve the problems of high processing pressure and playback stuttering caused by the synthesis of video by the user terminal in the prior art.

[0005] To address the aforementioned technical problems, this application provides a video synthesis method for a camera device. This method includes: acquiring a recorded video stream during video surveillance; the video parameters of the recorded video stream being lower than those of the surveillance video stream; extracting target video frames containing target objects from the recorded video stream to obtain a video library for editing; synthesizing a target video based on at least some of the target video frames from the video library; and sending the target video to a user terminal.

[0006] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a camera device, including a lens module, a processor, and a memory; the lens module and the memory are coupled to the processor, the memory is used to store program instructions, and the processor is used to execute the program instructions to implement the video synthesis method of the camera device provided in this application as described above, including: acquiring and recording a video stream during video monitoring; the video parameters of the recorded video stream are lower than the video parameters of the monitoring video stream; extracting target video frames containing target objects from the recorded video stream to obtain a video library for editing; synthesizing a target video based on at least some of the target video frames based on the video library; and sending the target video to a user terminal.

[0007] The beneficial effects of this application are as follows: Unlike the prior art, this embodiment acquires and records video streams during the video monitoring process, and the video parameters of the recorded video stream are lower than those of the monitoring video stream. Since this embodiment acquires and records video streams separately without affecting the video monitoring process, finds the video frame containing the target object in the recorded video stream, determines the target video frame containing the target object, and synthesizes at least some of the target video frames into a target video, it does not affect the user terminal's viewing of the video monitoring. Furthermore, by separately acquiring and using the recorded video stream in the camera device to determine the target video frame, synthesizing the target video using at least some of the target video frames, and finally sending the target video to the user terminal for viewing, it eliminates the need for the user terminal to perform video synthesis online using the footage extracted from the video monitoring. This reduces the significant processing pressure on the user terminal caused by the complex steps involved in such video processing and synthesis, thereby saving resources, improving the smoothness of video monitoring playback, and also improving the smoothness of target video playback. In the above process, on the one hand, it does not require the use of other professional software for compositing, which helps to save compositing costs. On the other hand, by completing the video compositing on the camera equipment, it reduces the playback stuttering problem caused by the user terminal when creating video files, which can improve the smoothness of video file playback on the user terminal and reduce the probability of playback stuttering. Attached Figure Description

[0008] Figure 1 This is a schematic flowchart illustrating an embodiment of the video synthesis method for the camera equipment of this application;

[0009] Figure 2 This is a timing diagram illustrating an embodiment of the video synthesis method for the camera device of this application;

[0010] Figure 3 This is a schematic diagram illustrating the steps of the video synthesis method using the camera equipment described in this application;

[0011] Figure 4 This is a schematic diagram of the circuit structure of an embodiment of the camera device of this application;

[0012] Figure 5 This is a schematic diagram of the circuit structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] Video production is the process of re-editing, integrating, and arranging images, videos, and background music to generate a new video file. It's not only about compositing the original materials but also about reprocessing them. Video compositing is a crucial step in video production, involving splicing processed or edited images or videos according to certain rules to create a new image or video. Video compositing is frequently used in film and television production. It involves shooting necessary segments in different scenes, then compositing these segments according to the script's order, adding voice-over and background music, and finally presenting the audience with a continuous sequence of movie or television scenes.

[0015] Through long-term research, the inventors of this application have discovered that in related technologies, users can use video recording devices, such as surveillance cameras, smart home cameras, and smart pet cameras, to capture images on user terminals, such as computers, mobile phones, and tablets, to create video diaries of monitored content. However, if they want to use AI functions to recognize certain images and synthesize them to create video diaries, they generally need to use specialized software. For example, if they want to synthesize video footage of pets from a camera on a pet feeder to create a pet log, they often need to use a corresponding app on the user terminal to synthesize and edit the video footage of pets captured by the camera. This not only leads to excessive consumption of processing resources on the user terminal, excessive processing pressure, and lag, which in turn causes playback stuttering, but also increases the production cost of synthesized videos. To solve the above technical problems, this application proposes the following embodiments.

[0016] The following embodiments of this application can be applied to camera devices, which can be standalone devices such as cameras, or cameras mounted on other mobile terminals, such as cameras on pet feeders, pet water fountains, surveillance cameras, etc. During use, the camera device is powered on and initialized.

[0017] The camera device can synthesize the corresponding target video and send it to the user terminal. The user terminal can be a terminal that has pre-established point-to-point communication with the camera device, such as a mobile phone, tablet, or computer. Furthermore, the user terminal can also have a dedicated APP installed for connecting to the camera device. When initially connecting the camera device to the user terminal, the user can use the APP to scan a code with the camera device to obtain its device information, including device type, device name, and device serial number (SN). This information is then reported to the APP server for registration via a pre-set network module. The user can then obtain video data, view, and download the target video through the APP on the user terminal.

[0018] like Figure 1As shown, an embodiment of the video synthesis method for the camera device of this application may include: S100: Acquiring a recorded video stream during video monitoring by acquiring a surveillance video stream; the video parameters of the recorded video stream are lower than those of the surveillance video stream. S200: Extracting target video frames containing the target object from the recorded video stream to obtain a video library for editing. S300: Synthesizing a target video based on at least some of the target video frames from the video library. S400: Sending the target video to the user terminal.

[0019] By acquiring and recording video streams during video surveillance, and ensuring that the video parameters of the recorded video stream are lower than those of the surveillance video stream, this embodiment achieves this without affecting the video surveillance process. The recorded video stream identifies the video frame containing the target object and determines the target video frame. At least some of these target video frames are then combined to form the target video. This approach avoids impacting the user terminal's viewing experience of the video surveillance. Furthermore, by separately acquiring and using the recorded video stream to determine the target video frame in the camera device, and then combining at least some of these target video frames to form the target video, the target video is finally sent to the user terminal for viewing. This eliminates the need for the user terminal to perform video synthesis online using the footage extracted from the surveillance video, reducing the processing pressure on the user terminal caused by the complex steps involved in video processing and synthesis. This saves resources, improves the smoothness of the video surveillance playback, and also enhances the smoothness of the target video playback. In the above process, on the one hand, it does not require the use of other professional software for compositing, which helps to save compositing costs. On the other hand, by completing the video compositing on the camera equipment, it reduces the playback stuttering problem caused by the user terminal when creating video files, which can improve the smoothness of video file playback on the user terminal and reduce the probability of playback stuttering.

[0020] The following describes this embodiment in detail. This embodiment may include:

[0021] S100: During the process of acquiring and recording video streams for video monitoring, the video parameters of the recorded video stream are lower than those of the monitoring video stream.

[0022] Video surveillance refers to the use of cameras to observe the environment within a certain range. While the camera is operational, it outputs a monitoring video stream and sends it to a user terminal. Users can view the video stream in real-time on their terminals, thus understanding the environment within the camera's monitoring range. In some implementations, the monitoring video stream is simultaneously stored locally on the camera or on the user terminal (e.g., a computer or mobile phone) while being sent to the user terminal. Users can then retrieve and play back previous videos. In other implementations, the camera simultaneously outputs a playback video stream, which is stored on the camera or on the user terminal for easy playback.

[0023] A recorded video stream can be a video stream output separately from the video surveillance process. The video parameters of the recorded video stream are lower than those of the surveillance video stream. The recorded video stream can be the video stream captured by the camera after initialization, or it can be the video stream captured when a preset event is triggered, such as when a target object enters the camera's frame. In other words, the recorded video stream can be captured separately by the camera in addition to capturing the surveillance video stream. The target object can be a human body, an animal body, other movable objects, or inanimate objects. The recorded video stream is used for subsequent target video production.

[0024] For instructions on how to capture and record video streams, please refer to the following steps included in S100:

[0025] S110: Simultaneously acquire and record video streams and detection video streams during the acquisition of monitoring video streams.

[0026] The detection video stream is used to detect and identify objects in the video frame. For example, the detection video stream can be an unencoded video stream specifically designed for object detection and recognition, such as a YUV format video stream. Since this embodiment requires the identification of a target object, using an unencoded detection video stream allows for faster and more efficient identification of the target object.

[0027] The recorded video stream and the detected video stream are captured synchronously on the same camera device, meaning that the corresponding video frames are consistent and the frame sequences can also be synchronized.

[0028] In one implementation, S110 may include the following steps:

[0029] S111: Determine whether the target object has been identified in the monitoring screen corresponding to the monitoring video stream.

[0030] Once the camera is turned on, it will continuously output a monitoring video stream. For example... Figure 2As shown, users can monitor the situation within the camera's range via a live video stream on their user terminals, thus achieving real-time monitoring. When the camera is off, no live video stream is generated, meaning video monitoring is impossible. Furthermore, the video parameters of the recorded video stream can be lower than those of the live video stream. Specifically, the bitrate of the recorded video stream can be lower than that of the live video stream. By setting the video parameters of the recorded video stream to be lower than those of the live video stream, the file size of the video file synthesized using the recorded video stream is smaller. Compared to directly using the live video stream for synthesis, using the recorded video stream for synthesis achieves a smoother transmission and playback effect when synthesizing a video of a certain length.

[0031] For example, the video parameters of a typical surveillance video stream output by a camera device when it is turned on are: 1080p resolution, 15fps frame rate, GOP (keyframe period) of 30, and bitrate of 200kbps. However, the video parameters of the recorded video stream in this embodiment are: 480p resolution, 6fps frame rate, GOP of 24, and bitrate of 200kbps. Therefore, by changing the video parameters of the recorded video stream in this embodiment, a smaller video size can be achieved when synthesizing a video of a certain duration, thus facilitating smooth transmission and reducing the probability of playback stuttering on the user's terminal.

[0032] Furthermore, the parameters of the recorded video stream can be dynamically configured according to the device's capabilities. In some implementations, the user can change various video parameters on the user terminal's configuration page for the camera device, such as changing the video parameters of the recorded video stream, or controlling other terminals of the camera device to change video parameters. For example, when the user needs to synthesize a short video with high clarity and good image quality, the resolution and GOP value in the video parameters can be increased to obtain a higher quality video image. When the user needs to synthesize a longer video, the resolution and GOP value in the video parameters can be decreased to reduce the space occupied by the synthesized video and increase the length of the synthesized video. In other implementations, after the camera device and the user terminal are connected, the camera device can read the user terminal's hardware configuration information and match appropriate video parameters for the user terminal, such as changing the video parameters of the recorded video stream, to achieve intelligent dynamic matching.

[0033] The system analyzes the surveillance video stream to determine if a target object is detected, thus informing whether recording should begin. Target objects can include, for example, human faces, human figures, dog faces, or dog figures.

[0034] Since surveillance video streams begin recording and outputting as soon as the camera starts, they can continue recording and outputting regardless of whether objects exist in the monitored area or whether the monitored area changes. If recording and detection video streams were also acquired from the beginning, it would result in high power consumption for the camera equipment. Therefore, determining whether the monitored area corresponding to the surveillance video stream has detected a target object is used as a preset trigger event for whether to acquire recording and detection video streams.

[0035] Specifically, in the process of judging the monitoring screen corresponding to the surveillance video stream, if the target object is a living person or a movable object, it can be determined by judging whether there is a change in the screen between two consecutive frames. If there is a change in the screen between two consecutive frames, it can be assumed that a living person has moved. If the target object is a face, it can be determined whether the target object exists in the monitoring screen through face recognition. When a face is recognized, it means that the target object exists in the monitoring screen.

[0036] S112: If so, then start collecting and recording video streams and detecting video streams simultaneously.

[0037] When a target object is present in the monitored screen corresponding to the surveillance video stream, simultaneous acquisition and recording of the video stream and detection of the video stream will begin. In other words, after the camera is turned on, it can continuously output a surveillance video stream, but when a target object is detected in the monitored screen corresponding to the surveillance video stream, simultaneous acquisition and recording of the video stream and detection of the video stream will begin. For example, if the target object is a dog, the acquisition and recording of the video stream and detection of the video stream will begin the moment the dog appears in the monitored screen.

[0038] If not, meaning the target object is not identified in the monitoring screen corresponding to the monitoring video stream, the judgment on whether the target object is identified will continue until the target object is identified, at which point the acquisition and recording of the video stream and the detection video stream will begin.

[0039] S200: Extract the target video frame containing the target object from the recorded video stream to obtain a video library for editing.

[0040] The target video frame can be one or more frames containing the target object in the recorded video stream. Since the recorded video stream can start capturing after the target object is detected, and this embodiment is a video synthesis for the target object, after identifying the video frames containing the target object in the recorded video stream, these video frames are used as the target video frames for synthesis.

[0041] A video library can be a collection of target video frames. During editing, several target video frames are extracted from the video library to synthesize the target video.

[0042] For instructions on how to extract target video frames containing target objects from a recorded video stream to obtain a video library for editing, please refer to the following steps included in S200:

[0043] S210: Identify and detect video frames in the video stream that contain the target object.

[0044] The detection video stream can be in YUV format and is unencoded. Encoding can be based on the H.264 video coding standard or other encoding standards. Since encoded video streams may degrade image quality and reduce accuracy in target object recognition, this embodiment uses an unencoded detection video stream for identification. This allows for more accurate identification of the video stream containing the target object, improving recognition accuracy.

[0045] Since the detection video stream is composed of several video frames arranged in a time sequence, some of these frames may capture the target object, while others may capture only a portion of the target object or none at all. Target object identification is then performed sequentially on the acquired video stream to identify the detection video frames containing the target object.

[0046] In one implementation, S210 may include the following steps:

[0047] S211: Perform target object recognition on the video frames already acquired in the detection video stream to identify the detection video frames and obtain the tag information corresponding to the detection video frames.

[0048] Target object recognition refers to identifying each frame in a detection video stream to determine whether a target object exists in each frame. During the acquisition of the detection video stream, the stream is output frame by frame, and the camera device can perform target object recognition on each frame.

[0049] If the target object includes a human face, then face recognition is performed on the video frames already captured in the detection video stream. If the target object includes a pet, then pet recognition is performed on the video stream already captured in the detection video stream.

[0050] The tagging information can be used to identify the detected video frame containing the target object. The tagging information is equivalent to the identity information of the identified detected video frame, facilitating the distinction between the detected video frame and other video frames, and also distinguishing different detected video frames. For how to obtain the tagging information corresponding to the detected video frame containing the target object, please refer to the following steps included in S211:

[0051] S2110: Mark the timestamp, object type, and position data of the target object in the detected video frame to obtain the marking information corresponding to each detected video frame.

[0052] Target object recognition is based on pre-determined target objects and is performed using an AI recognition model built into the camera device. This AI recognition model can be updated via network connection or through a user terminal connected to it. The AI ​​recognition model can be a collection of multiple pre-trained neural network recognition models. These models can include human figures, faces, pets, etc. For example, the pet model can include recognition models of the face, posture, and voice of cats and dogs. The AI ​​recognition model can also be extended with other AI detection functions. When the target object is a dog, the target object is considered identified when the AI ​​recognition model detects a dog shape within each frame of the detection video stream.

[0053] The identification information may include the timestamp of the detected video frame, the object type of the target object in the detected video frame, and the location data of the target object in the detected video frame.

[0054] A timestamp is a sequence of characters or encoded information used to identify when a specific event occurred, typically providing a date and time. In this embodiment, the timestamp could be the specific time corresponding to the detected video frame containing the target object.

[0055] The object type of the target object can be determined based on the type in the AI ​​recognition model when a detection video frame containing the target object is detected. In other words, when the AI ​​recognition model identifies the target object by recognizing a dog shape, the object type of the target object is a dog shape. For example, the object type can include a face, a human shape, and a pet.

[0056] The position data of the target object within the detected video frame. For example, the position data can be obtained by establishing a coordinate system within the frame corresponding to the detected video frame and determining the coordinates of the area where the target object is located, or it can be obtained based on other methods for determining position.

[0057] During the AI ​​recognition process of each frame in the detected video stream, when a target object is detected in one or more frames, the timestamp, object type, and position of the target object within each frame are recorded to form the tagging information for each frame. For example, if the target object is a dog, and the target object is detected in the video frame at 1 minute and 30 seconds, the AI ​​recognition model will output the tagging information for that frame as follows: timestamp 1 minute and 30 seconds, target object is a dog, and the target object's position is determined to be (6,6) based on coordinates.

[0058] After identifying the detection video frame containing the target object, the target video frame can be determined in the recorded video stream using the detection video frame. See the following steps included in S200:

[0059] S220: Extract the video frame corresponding to the detected video frame from the recorded video stream and use it as the target video frame.

[0060] Since the recording and detection video streams are captured synchronously, and both streams capture the same scene, the visual presentation is identical. Therefore, the video frames in the recording and detection streams can be synchronous and corresponding. In other words, when a target object is detected in a frame of the detection video stream, the corresponding frame in the recording video stream will also contain the same target object. Because the detection video stream is unencoded, it cannot be directly used for video compositing; doing so would significantly consume the camera's processing resources. Therefore, in the dedicated video stream used to generate the recording (the recording video stream), the video frame corresponding to the detection video frame containing the target object can be identified and used as the target video frame for compositing.

[0061] In one implementation, S220 may include the following steps:

[0062] S221: Extract the video frame corresponding to the marking information from the video frames already captured in the recorded video stream, and use it as the target video frame.

[0063] Since the recorded and detected video streams are acquired synchronously, meaning both correspond to the same video frame, there can be a one-to-one correspondence between video frames in both streams. This means each frame shares the same tagging information. Therefore, by analyzing the tagging information of the detected video frames, the corresponding video frame in the recorded video stream can be extracted, and this detected video frame must also correspond to the detected video frame with that tagging information. If the detected video frame corresponding to the tagging information contains a target object, then the video frame in the recorded video stream corresponding to that tagging information is taken as the target video frame.

[0064] Furthermore, regarding how to extract the video frames corresponding to the tagging information, please refer to the following steps included in S221:

[0065] S2210: Extract the video frames that match each timestamp from the video frames already captured in the recorded video stream, use them as target video frames, and store them to form a video library.

[0066] Since both the recorded and detected video streams capture the same video frame—meaning the captured frame is identical at every moment—the video frame in the recorded video stream that corresponds to the timestamp of the detected video frame containing the target object also contains the target object. This video frame is then stored as the target video frame, forming the video library. For example, if the timestamp of the detected video frame containing the target object in the detected video stream is 1 minute and 30 seconds, then the video frame corresponding to 1 minute and 30 seconds in the recorded video stream is the target video frame, and this target video frame is stored in the camera device's preset storage module, forming the video library.

[0067] Furthermore, during the process of storing target video frames in the video library, the stored target video frames can be categorized, for example, according to the object type of the target object. For instance, when the target object is a dog, all detected target video frames will contain a dog shape. Since the object type corresponding to a dog shape is "dog," all target video frames with a dog shape can be stored in the group for "dog" object type. Similarly, when the target object is a human face, all detected target video frames will contain a human face. Since the object type corresponding to a human face is "human," all target video frames with a human face can be stored in the group for "human" object type. Categorizing and storing target video frames facilitates the differentiation and management of target video frames in the video library, thereby improving the efficiency of extracting different types of target video frames from the video library.

[0068] S300: Synthesizes a target video from at least some of the target video frames based on a video library.

[0069] The target video can be a video file obtained by compositing target video frames that at least partially contain the target object. The target video frames used to compose the target video can be all target video frames containing the target object identified in the recorded video stream, or a subset of all target video frames containing the target object identified in the recorded video stream.

[0070] For instructions on how to synthesize at least some of the target video frames into a target video, please refer to the following steps included in S300:

[0071] S310: Extract at least a portion of the target video frames from the video library.

[0072] Once the target video frames are identified, they are all stored in the video library within the camera device's pre-installed storage module. When it's necessary to synthesize the target video frames into a single target video, they must first be extracted from the video library. Furthermore, during the extraction process, all target video frames can be extracted from the video library, or only a portion of them can be extracted. The specific number of target video frames to be extracted can be determined based on the configuration of the target video frames.

[0073] In one implementation, for how to extract at least a portion of the target video frames from the determined target video frames, see the following steps included in S310:

[0074] S311: Obtain synthesis parameters.

[0075] Combination parameters can be parameters used to indicate the extraction of target video frames and how the target video is combined. Combination parameters can be default parameters within the camera device, or they can be configured by the user through a user terminal, such as an app installed on the user terminal to control the camera device.

[0076] S312: Extract at least a portion of the target video frames from the video library that match the synthesis parameters.

[0077] After obtaining the synthesis parameters, the target video frame that matches the synthesis parameters can be selected during the extraction of the target video frame, and then the selected target video frame can be synthesized.

[0078] Synthesis parameters may include preset object type, generation time period, preset duration, audio data, and preset special effects data.

[0079] In one implementation, the composition parameters include a preset object type, and S312 may include the following steps:

[0080] S3120: Extract at least a portion of the target video frames from the video library whose object type matches a preset object type.

[0081] The preset object type is a pre-defined object type, which is the type of object that the target video will be used for compositing. For example, if the preset object type is "dog," then target video frames with the object type "dog" will be extracted from the target video frames. At least some of these extracted target video frames can be used for compositing.

[0082] In one implementation, the synthesis parameters include the generation time period, and S312 may include the following steps:

[0083] S3121: Extract at least a portion of the target video frames located within the generation time period from the video library.

[0084] The generation time period can be a parameter used to limit the generation time of target video frames. In other words, during the extraction of target video frames, only those captured / generated within the generation time period are extracted; target video frames from other time periods are not extracted. For example, if the generation time period is from 8 AM to 8 PM, then during the extraction of target video frames, only target video frames generated in the video library during this period will be extracted; target video frames generated before 8 AM and after 8 PM will not be included in the extraction.

[0085] In one implementation, when the synthesis parameters include a preset duration, step S312 may include the following steps:

[0086] S3122: Extract at least a portion of the target video frames from the video library according to the number required for the preset duration, such that the number of extracted target video frames is the same as the number required for the preset duration.

[0087] The preset duration can be the length of the target video. This preset duration can be the default value within the camera device or configured by the user through a pre-configured app. When a user wants to synthesize a target video of a certain duration, a sufficient number of target video frames need to be extracted to meet the synthesis requirements within the preset duration. For example, if the recorded video stream has a frame rate of 6fps, and the user wants to synthesize a target video with a preset duration of 60 seconds, since the frame rate of the video stream refers to the number of frames transmitted per second, 60 × 24 = 1440 target video frames need to be extracted to meet the required number of target video frames for the preset duration.

[0088] S320: Perform composite processing on the extracted target video frames to generate the target video.

[0089] Synthesis processing can be achieved by piecing together the extracted target video frames in a certain order to form the target video.

[0090] For instructions on how to synthesize the extracted target video frames to generate the target video, please refer to the following steps included in S320:

[0091] S3200: Uses preset effects data to perform effects processing on the target objects in the extracted target video frames.

[0092] In one implementation, the compositing parameters may also include preset effects data. Preset effects data can be data used to give video images special effects, such as special effects images. After extracting a target video frame that conforms to other compositing parameters, effects processing can be applied to the target object in the extracted target video frame based on the preset effects data. The camera device can update the preset effects data stored within it, for example, through network updates or user terminal configuration.

[0093] For example, the preset special effects data is to add rabbit ears to the target object. When the target video frame is extracted and the target object is a dog's face, the rabbit ears effect is added to the dog's face, thus achieving the special effects processing of the target object.

[0094] S3201: Combine the target video frames after special effects processing to generate the target video.

[0095] After the target video frames are processed with special effects, they become target video frames with special effects. Then, these target video frames with special effects are composited, which means splicing them together in a certain order. When splicing, they can be spliced ​​according to the time when the target video frames were acquired, or according to the order defined by the user, so as to obtain the target video with special effects.

[0096] like Figure 3 As shown, in this embodiment, after simultaneously starting the acquisition of the detection video stream and the recording video stream, an AI recognition model is used to identify target objects in the detection video stream, and the detection video frames containing target objects are marked. Based on the marking information, the target video frames in the recording video stream corresponding to the detection video frames containing target objects are determined and stored in the storage module. When compositing is required, the target video frames are extracted from the storage module and sent to the video compositing module to splice and combine the target video frames, thereby obtaining the target video.

[0097] Following S320, the following steps may also be included:

[0098] S330: Insert audio data into the target video and send the target video with the inserted audio data to the user terminal.

[0099] Furthermore, the compositing parameters can also include audio data, which can be background music to enhance the auditory experience of the target video, or audio clips added to better fit the scene in the target video. After the target video is obtained through compositing, the audio data is inserted into the target video to improve its viewing experience.

[0100] Specifically, in the process of inserting audio data into the target video, since the audio data is composed of several audio frames, inserting the audio data into the target video means inserting the audio frames into the target video frames, thereby enabling the audio to accompany the playback of the target video.

[0101] S400: Sends the target video to the user terminal.

[0102] After synthesizing the target video frames based on the synthesis parameters to obtain the target video, the obtained target video can be sent to the user terminal, and the user can watch the target video through the user terminal.

[0103] In related technologies, video synthesis using video streams captured by camera equipment via user terminals may present the following problems:

[0104] On the one hand, if the target is an animal, the user terminal will have low recognition efficiency through the APP, and it will be difficult to set up an AI recognition model for the target within the APP.

[0105] The camera device in this embodiment can be equipped with an AI recognition model for the target object, which can improve the recognition rate of the target object, thereby improving the accuracy and matching degree of special effects processing.

[0106] On the other hand, if a user terminal performs video synthesis processing using monitoring video streams, playback video streams, or recorded video streams via remote communication, it cannot adjust or control the frame rate and bitrate of the video, resulting in a very large generated video. Since the generated video is stored on the camera device and playback is achieved through the communication protocol between the device and the user terminal, if the video is too large, it will cause playback stuttering on the user terminal.

[0107] In this embodiment, the recorded video stream is acquired separately during monitoring. The target video frames are identified in the recorded video stream using the camera device, and then a portion of these target video frames are used to synthesize the target video. This approach avoids interfering with the user terminal's video monitoring and eliminates the need for video synthesis on the user terminal, reducing its processing load. The synthesized target video is then sent to the user terminal, allowing for smooth playback without requiring online video synthesis via communication protocols.

[0108] On the other hand, user terminals need to add special effects when playing videos. During playback, the app detects the target object and then adds effects to it. If the specific object is an animal, the app's recognition rate is low, and there is a delay because it needs to be recognized first and then the effects added.

[0109] In this embodiment, after the target video frame is determined, at least a portion of the target video frames can be extracted from the stored target video frames. Precise special effects data is used to accurately and quickly add special effects to the extracted at least a portion of the video frames. There is no pressure from network latency or limitation of communication protocols. Moreover, since the target video frames have been determined, special effects processing of the target video frames can be performed concurrently, resulting in faster processing speed.

[0110] In summary, in the video method of the camera device in this embodiment, the camera device continuously outputs a monitoring video stream after being turned on. When a target object is detected in the monitoring screen corresponding to the monitoring video stream, the recording video stream and the detection video stream are simultaneously acquired. Since the detection video stream is an unencoded video stream with no loss of image quality, target object identification is performed on each frame in the detection video stream to obtain a detection video frame containing the target object. Since the detection video stream and the recording video stream acquire the same video screen at the same time, the target video frame corresponding to it in the recording video stream can be determined according to the timestamp of the detection video frame containing the target object, and the target video frame is stored in the storage module of the camera device. During the compositing process, compositing parameters are first obtained, and then target video frames that meet the compositing parameters are selected from the storage module. When the compositing parameters include preset special effects data, the selected target video frames are processed with special effects based on the preset special effects data. The target video frames with special effects are then composited to obtain the target video. Furthermore, when the compositing parameters also include audio data, the audio data is inserted into the target video. Finally, the target video with inserted audio data is sent to the user terminal, where the user can play and watch the target video. This allows the target video to be synthesized without the need for other professional software. Furthermore, by synthesizing the video using a camera device, even when synthesizing large videos, the video can be played smoothly on the user terminal, reducing the probability of playback stuttering.

[0111] For example, when the video compositing method of this camera device is applied to a pet feeder, video is captured by a camera installed on the pet feeder. If the user owns a dog and wants to create a video diary of the dog's activities, the target object can be pre-set as a dog shape. When the pet feeder is on, the camera captures a monitoring video stream. The user can view the monitoring video stream on a user terminal connected to the pet feeder for real-time monitoring. Simultaneously, the monitoring screen is continuously evaluated and detected. When the target object, i.e., a dog shape, appears in the monitoring screen, the capture and recording of the video stream and the detection video stream begin simultaneously. The AI ​​recognition model pre-installed in the pet feeder identifies dog shapes in each frame of the detection video stream, resulting in dog-shaped detection video frames. Since the detection video stream is unencoded and cannot be transmitted, target video frames corresponding to these dog-shaped detection frames need to be obtained from the recorded video stream. Because the recorded and detection video streams capture the same footage, the timestamps of the dog-shaped detection video frames determine the target video frames in the recorded video stream, which are then stored in a pre-installed storage module. When compositing the dog-shaped target video frames, the default or user-modified compositing parameters of the pet feeder are first obtained. Assuming the preset object type is dog, the generation time is from 8 AM to 8 PM, and the preset duration is 1 minute, with a preset frame rate of 6fps, 1440 dog-shaped target video frames generated between 8 AM and 8 PM need to be selected from the storage module. Furthermore, when the synthesis parameters include preset special effects data, it is also necessary to perform special effects processing on each target video frame with a dog shape, and then synthesize these 1440 target video frames into a target video. Finally, the synthesized target video is sent to the user terminal for the user to watch.

[0112] like Figure 4 As shown in the embodiments of the camera device described in this application, the camera device 100 includes a processor 110 and a memory 120. The memory 120 is coupled to the processor 110.

[0113] The memory 120 is used to store computer programs and may be RAM, ROM, or other types of storage devices. Specifically, the memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory is used to store at least one line of program code.

[0114] Processor 110 is used to control the operation of camera device 100. Processor 110 can also be referred to as CPU (Central Processing Unit). Processor 110 may be an integrated circuit chip with signal processing capabilities. Processor 110 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), image processor (GPU), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 110 can be any conventional processor.

[0115] The processor 110 is used to execute the computer program stored in the memory 120 to implement the video synthesis method described in the embodiments of the video synthesis method of the camera device of this application.

[0116] In some embodiments, the camera device 100 may further include a peripheral device interface 130 and at least one peripheral device. The processor 110, memory 120, and peripheral device interface 130 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 130 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 140, a lens module 150, an audio circuit 160, and a power supply 170.

[0117] Peripheral device interface 130 can be used to connect at least one I / O (Input / output) related peripheral device to processor 110 and memory 120. In some embodiments, processor 110, memory 120 and peripheral device interface 130 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 110, memory 120 and peripheral device interface 130 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0118] The radio frequency (RF) circuit 140 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 140 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 140 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 140 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 140 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 140 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0119] The lens module 150 may include a lens and an image sensor. The lens projects collected light onto the image sensor, converting the light signal into an electrical signal to capture an image. The lens may include several lens elements, on which light is projected to capture light. The image sensor utilizes the photoelectric conversion function of an optoelectronic device to convert the light image on the photosensitive surface into an electrical signal proportional to the light image. For example, the image sensor may be a CMOS or CCD.

[0120] The audio circuit 160 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 110 for processing, or input to the radio frequency circuit 140 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the camera device 100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 110 or the radio frequency circuit 140 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 160 may also include a headphone jack.

[0121] Power supply 170 is used to power the various components in camera device 100. Power supply 170 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 170 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0122] For a detailed description of the functions and execution processes of each functional module or component in the embodiments of the camera device of this application, please refer to the description in the above embodiments of the video synthesis method of the camera device of this application, which will not be repeated here.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed camera device and video synthesis method can be implemented in other ways. For example, the camera device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] See Figure 5If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium 200. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions / computer programs to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as imaging devices such as computers, mobile phones, laptops, tablets, and cameras that have the aforementioned storage media.

[0127] In summary, by acquiring and recording video streams during video surveillance, and ensuring that the video parameters of the recorded video stream are lower than those of the surveillance video stream, this embodiment achieves this without affecting the video surveillance process. It identifies the video frames containing the target object within the recorded video stream, determines the target video frames, and synthesizes at least some of these target video frames into a target video. This approach avoids impacting the user terminal's viewing experience of the video surveillance. Furthermore, by separately acquiring and using the recorded video stream to determine the target video frames in the camera device, and synthesizing the target video using at least some of these frames, the target video is finally sent to the user terminal for viewing. This eliminates the need for the user terminal to perform video synthesis online using the footage extracted from the video surveillance, reducing the processing pressure on the user terminal caused by the complex steps involved in video processing and synthesis. Consequently, it saves resources, improves the smoothness of video surveillance playback, and enhances the smoothness of target video playback. In the above process, on the one hand, it does not require the use of other professional software for compositing, which helps to save compositing costs. On the other hand, by completing the video compositing on the camera equipment, it reduces the playback stuttering problem caused by the user terminal when creating video files, which can improve the smoothness of video file playback on the user terminal and reduce the probability of playback stuttering.

[0128] The execution process of program data in a computer-readable storage medium can be described with reference to the video synthesis method embodiment of the camera device of this application described above, and will not be repeated here.

[0129] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A video synthesis method for a camera device, characterized in that, include: During the process of acquiring and recording video streams for video surveillance; The video parameters of the recorded video stream are lower than the video parameters of the monitored video stream; Extract the target video frames containing the target object from the recorded video stream to obtain a video library for editing; Based on the video library, at least some of the target video frames are synthesized into a target video; The target video is sent to the user terminal; The process of acquiring and recording video streams during video surveillance includes: simultaneously acquiring the recorded video stream and the detection video stream during the acquisition of the surveillance video stream, wherein the detection video stream is an unencoded video stream specifically used for object detection and recognition. The step of extracting the target video frame containing the target object from the recorded video stream includes: identifying the detection video frame containing the target object in the detection video stream; and extracting the video frame corresponding to the detection video frame from the recorded video stream as the target video frame.

2. The method according to claim 1, characterized in that: The step of identifying the detection video frames containing the target object in the detection video stream includes: Target object identification is performed on the video frames already acquired in the detection video stream to identify the detection video frames and obtain the tag information corresponding to the detection video frames; Extracting the video frame corresponding to the detected video frame from the recorded video stream as the target video frame includes: The video frame corresponding to the marker information is extracted from the video frames already captured in the recorded video stream and used as the target video frame.

3. The method according to claim 2, characterized in that: The step of obtaining the tag information corresponding to the detected video frame includes: The timestamp corresponding to the detected video frame, the object type of the target object, and the position data of the target object in the detected video frame are marked to obtain the marking information corresponding to each detected video frame.

4. The method according to claim 3, characterized in that: The step of extracting the video frame corresponding to the marker information from the video frames already captured in the recorded video stream, and using it as the target video frame, includes: The video frames that match each timestamp are extracted from the video frames already captured in the recorded video stream and used as the target video frames, which are then stored to form the video library.

5. The method according to claim 3, characterized in that: The process of synthesizing a target video from at least a portion of the target video frames based on the video library includes: Special effects processing is performed on the target object in the corresponding target video frame using preset special effects data and the location data; At least a portion of the target video frames that have undergone special effects processing are synthesized to generate the target video.

6. The method according to claim 1, characterized in that: The step of synthesizing at least a portion of the target video frames into a target video based on the video library includes: Extract at least a portion of the target video frames from the video library; The extracted target video frames are then synthesized to generate the target video.

7. The method according to claim 6, characterized in that: Extracting at least a portion of the target video frames from the video library includes: Obtain the synthesis parameters; Extract at least a portion of the target video frames from the video library that match the synthesis parameters.

8. The method according to claim 7, characterized in that: The synthesis parameters include a preset object type; the step of extracting at least some of the target video frames that match the synthesis parameters from the video library includes: Extract at least a portion of the target video frames from the video library whose object type matches the preset object type.

9. The method according to claim 7, characterized in that: The synthesis parameters include a generation time period; the step of extracting at least a portion of the target video frames that match the synthesis parameters from the video library includes: Extract at least a portion of the target video frames located within the generation time period from the video library.

10. The method according to claim 7, characterized in that: The compositing parameters include preset special effects data; the step of compositing at least a portion of the extracted target video frames to generate the target video includes: The target object in the extracted target video frame is processed with special effects using the preset special effects data; The target video frames, after special effects processing, are combined to generate the target video.

11. The method according to claim 7, characterized in that: The synthesis parameters include a preset duration; the step of extracting at least a portion of the target video frames that match the synthesis parameters from the video library includes: Extract at least a portion of the target video frames from the video library according to the number required for the preset duration, such that the number of extracted target video frames is the same as the number required for the preset duration; The step of synthesizing at least a portion of the extracted target video frames to generate the target video includes: The extracted target video frames are synthesized to generate the target video with a playback duration of the preset duration.

12. The method according to claim 7, characterized in that: The synthesis parameters include audio data; after the extraction of at least a portion of the target video frames for synthesis processing to generate the target video, the process includes: The audio data is inserted into the target video, and the target video with the inserted audio data is sent to the user terminal.

13. The method according to claim 1, characterized in that: The process of acquiring and monitoring video streams includes simultaneously acquiring the recorded video stream and the detected video stream, including: Determine whether the target object is identified in the monitoring screen corresponding to the monitoring video stream; If so, then the recording video stream and the detection video stream will be collected simultaneously.

14. A camera device, characterized in that, A lens module, a processor, and a memory; the lens module and the memory are coupled to the processor, the memory is used to store program instructions, and the processor is used to execute the program instructions to implement the video compositing method as described in any one of claims 1-13, comprising: During the process of acquiring and recording video streams for video surveillance, the video parameters of the recorded video stream are lower than those of the monitoring video stream. Extract the target video frames containing the target object from the recorded video stream to obtain a video library for editing; Based on the video library, at least some of the target video frames are synthesized into a target video; The target video is sent to the user terminal; The process of acquiring and recording video streams during video surveillance includes: simultaneously acquiring the recorded video stream and the detection video stream during the acquisition of the surveillance video stream, wherein the detection video stream is an unencoded video stream specifically used for object detection and recognition. The step of extracting the target video frame containing the target object from the recorded video stream includes: identifying the detection video frame containing the target object in the detection video stream; and extracting the video frame corresponding to the detection video frame from the recorded video stream as the target video frame.

Images