Video frame sharing output processing method and related device

Abstract

The invention discloses a video frame sharing output processing method and a related device, and relates to the technical field of software, and the method comprises the steps: obtaining an original video frame collected by a camera; the original video frame is converted into a first target video frame according to the format configuration of the stream pushing end, the original video frame is converted into a second target video frame according to the format configuration of the virtual camera, and the stream pushing end and the virtual camera are pre-created with respective shared memories; and writing the first target video frame into a shared memory of the stream pushing end, and writing the second target video frame into a shared memory of the virtual camera. According to the method, the architecture design of single-source processing and multi-target distribution can be realized, repeated calculation is avoided, in addition, cross-process access can be supported, and the effect of isolated storage of double shared memories is achieved.

Description

Technical Field

[0001] This application relates to the field of software technology, and in particular to a method and related apparatus for processing video frame sharing output. Background Technology

[0002] With the widespread adoption of applications such as live streaming and video conferencing, video data captured by cameras needs to simultaneously meet the dual requirements of streaming (cloud transmission) and front-end preview (viewing by local users). The processing of video frames by the streaming end and virtual cameras typically differs. Streaming ends usually require high resolution to ensure transmission quality, while virtual cameras, limited by the front-end display device, often support landscape mode with lower resolution.

[0003] Currently, the streaming client and the virtual camera each start their own processes to capture video frames, and each performs scaling and encoding processing independently. This results in the camera hardware resources being used repeatedly. Summary of the Invention

[0004] In view of the above problems, this application provides a method and related apparatus for processing video frame sharing output to solve the problem of asynchrony between the streaming end and the front-end preview screen. The specific solution is as follows:

[0005] The first aspect of this application provides a method for processing video frame sharing output, the method comprising:

[0006] Acquire the raw video frames captured by the camera;

[0007] The original video frame is converted into a first target video frame according to the format configuration of the streaming end, and the original video frame is converted into a second target video frame according to the format configuration of the virtual camera. The streaming end and the virtual camera are pre-created with their own shared memory.

[0008] The first target video frame is written to the shared memory of the streaming end, and the second target video frame is written to the shared memory of the virtual camera.

[0009] In one possible implementation, converting the original video frame into a first target video frame according to the format configuration of the streaming end includes:

[0010] Obtain the original resolution of the original video frame and the first target resolution in the format configuration of the streaming end;

[0011] If the original resolution is the same as the first target resolution, the original video frame is used as the first target video frame.

[0012] If the original resolution is different from the first target resolution, the original video frame is scaled using the original resolution as the initial value and the first target resolution as the target value to obtain the first target video frame.

[0013] In one possible implementation, converting the original video frame into a second target video frame according to the format configuration of the virtual camera includes:

[0014] Obtain the original resolution of the original video frame and the second target resolution in the format configuration of the virtual camera. The original resolution includes the original height and original width of the original video frame, and the second target resolution includes the target height and target width of the virtual camera.

[0015] If the original height of the original video frame is greater than its original width, the original vertical aspect ratio of the original video frame is determined based on the original height and original width of the original video frame.

[0016] The target height of the virtual camera is used as the target height of the original video frame, and the target width of the original video frame is determined based on the target height of the original video frame and the original aspect ratio of the vertical screen.

[0017] The original video frame is scaled using its original height and original width as initial values ​​and its target height and target width as target values ​​to obtain the second target video frame.

[0018] The black space width of the second target video frame is determined based on the target width of the original video frame and the target width of the virtual camera.

[0019] In one possible implementation, writing the second target video frame to the shared memory of the virtual camera includes:

[0020] Adjust the output component pointer of the shared memory of the virtual camera according to the black space width, and perform black space filling.

[0021] In one possible implementation, the step of converting the original video frame into a second target video frame according to the format configuration of the virtual camera further includes:

[0022] If the original height of the original video frame is less than its original width, compare the original height of the original video frame with the target height of the virtual camera, and compare the original width of the original video frame with the target width of the virtual camera.

[0023] If the original height of the original video frame is the same as the target height of the virtual camera, and the original width of the original video frame is the same as the target width of the virtual camera, then the original video frame is used as the second target video frame.

[0024] If the original height of the original video frame is different from the target height of the virtual camera, or if the original width of the original video frame is different from the target width of the virtual camera, the original video frame is scaled using the original height and original width of the original video frame as initial values ​​and the target height and target width of the virtual camera as target values ​​to obtain the second target video frame.

[0025] In one possible implementation, writing the first target video frame to the shared memory of the streaming end and writing the second target video frame to the shared memory of the virtual camera includes:

[0026] The first target video frame is written to the shared memory of the streaming end based on the mutex lock mechanism, and the second target video frame is written to the shared memory of the virtual camera.

[0027] A second aspect of this application provides a processing apparatus for video frame sharing output, the processing apparatus comprising:

[0028] The frame acquisition module is used to acquire the raw video frames captured by the camera.

[0029] A frame conversion module is used to convert the original video frame into a first target video frame according to the format configuration of the streaming end, and to convert the original video frame into a second target video frame according to the format configuration of the virtual camera. The streaming end and the virtual camera are pre-created with their own shared memory.

[0030] The frame writing module is used to write the first target video frame to the shared memory of the streaming end, and to write the second target video frame to the shared memory of the virtual camera.

[0031] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the video frame sharing output processing method described in the first aspect or any implementation thereof.

[0032] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program to enable the electronic device to implement the video frame sharing output processing method of the first aspect or any implementation thereof.

[0035] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform a video frame sharing output method as described in the first aspect or any implementation thereof.

[0036] By employing the above technical solution, this application provides a video frame sharing output processing method and related apparatus, comprising: acquiring raw video frames captured by a camera; converting the raw video frames into a first target video frame according to the format configuration of the streaming end; and converting the raw video frames into a second target video frame according to the format configuration of the virtual camera, wherein the streaming end and the virtual camera are pre-created with their respective shared memory; writing the first target video frame into the shared memory of the streaming end; and writing the second target video frame into the shared memory of the virtual camera. This application simultaneously converts a single video frame captured by the camera to generate the format required by both the streaming end and the virtual camera, and synchronously writes it into their respective shared memory. This enables a single-source processing and multi-target distribution architecture design, avoiding redundant calculations. Furthermore, the shared memory of the streaming end and the virtual camera is deployed independently, supporting cross-process access and achieving the effect of isolated storage with dual shared memory. Attached Figure Description

[0037] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0038] Figure 1 A flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application;

[0039] Figure 2 A partial flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application;

[0040] Figure 3 This is another part of the flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application;

[0041] Figure 4 This is another part of the flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application;

[0042] Figure 5A schematic diagram of a video frame sharing output processing device provided in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0045] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0047] To facilitate understanding of this application, the relevant concepts involved in this application are explained below:

[0048] Shared memory: an inter-process communication method that allows multiple processes to share the same physical memory through memory-mapped files.

[0049] Streaming end: refers to the module that encodes local video data and sends it to the cloud or live streaming platform.

[0050] Virtual camera: A camera device simulated by a driver or middleware, which allows the video stream to be used as a real camera by other applications (such as other software or web pages).

[0051] YUV420P Image Processing: Performs operations such as scaling, cropping, flipping, and black bar adjustment on the input YUV image data to adapt to different resolution or aspect ratio requirements.

[0052] Synchronization lock mechanism: Mutex locks are used to enable synchronized access to shared memory by multiple processes / threads, avoiding data read / write conflicts.

[0053] In existing live video applications, the streaming client and the virtual camera typically need to separately capture or process the same YUV420P format video source (supporting both landscape 16:9 and portrait 9:16 aspect ratios), leading to the following problems:

[0054] 1) Repeated scaling and format adaptation of video frames wastes CPU / GPU resources, especially in high-resolution video scenarios, where performance loss is significant.

[0055] 2) The streaming end and the virtual camera process data independently, which can easily lead to video synchronization delays (such as a frame difference between the streaming video and the front-end preview video), affecting the user experience.

[0056] 3) Virtual cameras only support 16:9 landscape resolution (such as 1280x720). In existing solutions, when adapting vertical video frames to virtual cameras, there is a lack of standardized black-out processing logic, which can easily lead to problems such as image stretching, aspect ratio distortion, or incorrect calculation of black-out areas.

[0057] To address the aforementioned problems, this application provides a method for processing video frame sharing output. The following detailed description of a video frame sharing output method according to an embodiment of this application is provided in conjunction with the accompanying drawings.

[0058] See Figure 1 , Figure 1 This is a flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application. Figure 1 As shown in the figure, the video frame sharing output processing method provided in this application embodiment may include steps S101 to S103, which are described in detail below.

[0059] S101, acquire the raw video frames captured by the camera.

[0060] In this embodiment, YUV420P video frames captured by a camera are received and used as raw data frames. These raw data frames serve as a single data source, avoiding repeated acquisition by multiple processes. Specifically, the YUV420P video frames from the camera can be uniformly received through an entry function, including the data length, width, and height of the YUV420P video frames.

[0061] S102, the original video frame is converted into a first target video frame according to the format configuration of the streaming end, and the original video frame is converted into a second target video frame according to the format configuration of the virtual camera. The streaming end and the virtual camera have their own shared memory pre-created.

[0062] In this embodiment of the application, the format configurations of the streaming end and the virtual camera can be obtained respectively. The original data frames are preprocessed by scaling, landscape and portrait adaptation, etc., according to the format configuration, so as to convert the original data frames into target video frames (i.e., the first target video frame) required by the inference end and target video frames (i.e., the second target video frame) required by the virtual camera.

[0063] In addition, this embodiment creates and manages two independent shared memory blocks, accessible to the streaming client and the virtual camera respectively, to achieve cross-process data sharing and avoid data write conflicts between different processes. In practical applications, memory-mapped files (i.e., shared memory) for the streaming client and the virtual camera can be created using the Windows API CreateFileMapping. The mapping size is the total size of a YUV420P video frame under the respective format configurations of the streaming client and the virtual camera (width × height × 3 / 2 bytes), and the mapping name is a preset unique identifier (such as "LPS_Stream_Share_Mem" or "VirCam_Share_Mem"), ensuring that the streaming client and the virtual camera can open the corresponding shared memory through the mapping name.

[0064] In one possible implementation, when converting the original data frame into the first target video frame required by the streaming end, the original resolution of the original video frame is compared with the target resolution (i.e., the first target resolution) in the format configuration of the streaming end. If they match, they can be copied directly; otherwise, scaling is used to adapt the format required by the streaming end. See also Figure 2 , Figure 2 This is a partial flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application. Figure 2 As shown in the embodiment of this application, a video frame sharing output processing method is provided. In step S102, "converting the original video frame into a first target video frame according to the format configuration of the streaming end" may include steps S201 to S203. These steps are described in detail below.

[0065] S201, obtain the original resolution of the original video frame and the first target resolution in the format configuration of the streaming end.

[0066] In this embodiment, the corresponding original resolution (including the original height and original width of the original video frame) can be obtained based on the original height and original width of the original video frame. In addition, the target height and target width required by the streaming end are determined according to the format configuration of the streaming end, thereby obtaining the first target resolution corresponding to the streaming end.

[0067] S202, if the original resolution is the same as the first target resolution, the original video frame is used as the first target video frame.

[0068] In this embodiment of the application, if the original resolution is the same as the first target resolution, the original video frame is used as the first target video frame.

[0069] S203: When the original resolution is different from the first target resolution, the original video frame is scaled using the original resolution as the initial value and the first target resolution as the target value to obtain the first target video frame.

[0070] In this embodiment of the application, if the original resolution is different from the first target resolution, the original resolution is used as the initial value and the first target resolution is used as the target value. The I420Scale interface of the libyuv library is called to scale the original video frame to obtain the first target video frame.

[0071] In one possible implementation, to address the limitation that virtual cameras only support landscape (e.g., 16:9) aspect ratios, the aspect ratio of the original video frames can be determined to indicate whether it is landscape or portrait. If the original data frame is portrait, the scaling width (ensuring a 16:9 ratio) is first calculated based on the height of the virtual camera, and then black areas are calculated and filled on the left and right sides to ensure the output is a standard landscape format. See also Figure 3 , Figure 3 This is another schematic flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application. For example... Figure 3 As shown in the embodiment of this application, a video frame sharing output processing method is provided. In step S102, "converting the original video frame into a second target video frame according to the format configuration of the virtual camera" may include steps S301 to S305. These steps are described in detail below.

[0072] S301, obtain the original resolution of the original video frame and the second target resolution in the format configuration of the virtual camera. The original resolution includes the original height and original width of the original video frame, and the second target resolution includes the target height and target width of the virtual camera.

[0073] In this embodiment, the corresponding original resolution (including the original height and original width of the original video frame) can be obtained based on the original height and original width of the original video frame. In addition, the target height and target width required by the virtual camera are determined according to the format configuration of the virtual camera, thereby obtaining the second target resolution corresponding to the virtual camera.

[0074] S302, when the original height of the original video frame is greater than its original width, determine the original vertical screen ratio of the original video frame based on the original height and original width of the original video frame.

[0075] In this embodiment of the application, if the original height of the original video frame is greater than its original width, the original video frame can be determined to be a portrait screen. Then, the original portrait screen ratio is determined based on the original height and original width of the original video frame. For example, if the original height of the original video frame is 720 and the original width is 405, then its original portrait screen ratio is 405 / 720=9 / 16.

[0076] S303 uses the target height of the virtual camera as the target height of the original video frame, and determines the target width of the original video frame based on the target height of the original video frame and the original aspect ratio of the portrait screen.

[0077] In this embodiment, the target height of the virtual camera is used as the target height of the original video frame, and the target width of the original video frame is determined by the target height of the original video frame and the original vertical screen ratio. That is, the target width of the original video frame = the target height of the original video frame × the original vertical screen ratio.

[0078] S304: Using the original height and original width of the original video frame as initial values ​​and the target height and target width of the original video frame as target values, scale the original video frame to obtain the second target video frame.

[0079] In this embodiment, the original height and width of the original video frame are used as initial values, and the target height and target width of the original video frame are used as target values. The I420Scale interface of the libyuv library is called to scale the original video frame to obtain the second target video frame.

[0080] S305, determine the black space width of the second target video frame based on the target width of the original video frame and the target width of the virtual camera.

[0081] In this embodiment, the black area width of the left and right sides of the second target video frame can be determined based on the target width of the original video frame and the target width of the virtual camera. The black areas on the left and right sides are symmetrical. Specifically, the black area width = (target width of the virtual camera - target width of the original video frame) / 2.

[0082] In one possible implementation, when the original data frame is in landscape mode, the original resolution of the original video frame can be compared with the second target resolution. If they match, it can be copied directly; otherwise, scaling is used to adapt the virtual camera to the required format. See also Figure 4 , Figure 4 This is another schematic flowchart illustrating a video frame sharing output processing method provided in an embodiment of this application. For example... Figure 4As shown in the embodiment of this application, a video frame sharing output processing method is provided. In step S102, "converting the original video frame into a second target video frame according to the format configuration of the virtual camera", it may also include steps S401 to S403. These steps are described in detail below.

[0083] S401, when the original height of the original video frame is less than its original width, compare the original height of the original video frame with the target height of the virtual camera, and compare the original width of the original video frame with the target width of the virtual camera.

[0084] In this embodiment of the application, if the original height of the original video frame is less than its original width, it can be determined that the original video frame is in landscape mode. Then, the original height of the original video frame is compared with the target height of the virtual camera, and the original width of the original video frame is compared with the target width of the virtual camera.

[0085] S402, if the original height of the original video frame is the same as the target height of the virtual camera and the original width of the original video frame is the same as the target width of the virtual camera, then the original video frame is used as the second target video frame.

[0086] In this embodiment of the application, if the original height of the original video frame is the same as the target height of the virtual camera and the original width of the original video frame is the same as the target width of the virtual camera, that is, the original resolution is the same as the second target resolution, then the original video frame is used as the second target video frame.

[0087] S403, if the original height of the original video frame is different from the target height of the virtual camera, or the original width of the original video frame is different from the target width of the virtual camera, the original video frame is scaled using the original height and original width of the original video frame as initial values ​​and the target height and target width of the virtual camera as target values ​​to obtain the second target video frame.

[0088] In this embodiment of the application, if the original height of the original video frame is different from the target height of the virtual camera, or the original width of the original video frame is different from the target width of the virtual camera, that is, the original resolution is different from the second target resolution, then the original height and original width of the original video frame are used as initial values, and the target height and target width of the virtual camera are used as target values. The I420Scale interface of the libyuv library is called to scale the original video frame to obtain the second target video frame.

[0089] S103, write the first target video frame to the shared memory of the streaming end, and write the second target video frame to the shared memory of the virtual camera.

[0090] In this embodiment, the first target video frame is directly written to the shared memory of the streaming end using mempy. Additionally, the second target video frame is directly written to the shared memory of the virtual camera using mempy. In practical applications, a starting pointer to the shared memory can be obtained using MapViewOfFile for subsequent video frame data writing. When the streaming end and the virtual camera exit, the shared memory mapping is released and the memory-mapped file handle is closed to prevent memory leaks.

[0091] In one possible implementation, when the original video frame is in portrait mode, its corresponding second target video frame can be written to the middle position of the shared memory of the virtual camera. This is specifically achieved by adjusting the output components and filling the black areas on the left and right sides with black. In this regard, the present application provides a method for processing shared video frame output, wherein step S103, "writing the second target video frame into the shared memory of the virtual camera," may include the following steps:

[0092] Adjust the output component pointer of the shared memory of the virtual camera according to the black space width, and perform black space filling.

[0093] In this embodiment, the output component pointer of the shared memory of the virtual camera is adjusted according to the blackout width. The output component pointer includes a Y component pointer, a U component pointer, and a V component pointer. Assuming the blackout width is left_pad, the Y component pointer is written starting from left_pad, the U component pointer is written starting from left_pad / 2 (the U component resolution is 1 / 2 of the Y component resolution), and the V component pointer is written starting from left_pad / 2 (the V component resolution is 1 / 2 of the Y component resolution).

[0094] Assume the target width of the virtual camera is vircam_Width, the target height of the virtual camera is vircam_Height, and the target width of the original video frame is scaled_width. When performing black padding on the Y component, we can iterate through the target height of the virtual camera and use memset to fill each row of Y component data with 0 bytes from left_pad to left_pad-1 bytes and from left_pad + scaled_width to vircam_Width-1 bytes on the right (Y component 0 represents pure black, ensuring that the black area has no brightness).

[0095] Additionally, since the resolution of the U / V component is half that of the Y component, when performing black padding for the U / V component, the height of the U / V component (0 to vircam_Height / 2-1) can be iterated. For each line of U / V component data, memset is used to fill the left side (0 to (left_pad / 2)-1 bytes) and the right side (left_pad / 2)+(scaled_width / 2) to (vircam_Width / 2)-1 bytes with 128 (in YUV420P format, 128 for the U / V component represents no color difference, avoiding color cast in the black area).

[0096] In one possible implementation, to ensure consistency between the data read by the streaming end and the virtual camera, a mutex lock mechanism can be used to ensure the atomicity of video frame data writing to shared memory, avoiding multi-process read / write conflicts. In this regard, an embodiment of this application provides a video frame shared output processing method, wherein step S103, "writing the first target video frame to the shared memory of the streaming end, and writing the second target video frame to the shared memory of the virtual camera," can be implemented using the following steps:

[0097] The first target video frame is written to the shared memory of the streaming end based on the mutex lock mechanism, and the second target video frame is written to the shared memory of the virtual camera.

[0098] In this embodiment, the shared memory of the streaming end and the shared memory of the virtual camera are managed independently using mutex locks to avoid the streaming end's write blocking affecting the virtual camera's write, or the virtual camera's write blocking affecting the streaming end's write, thereby improving concurrent processing efficiency.

[0099] Specifically, when the first target video frame is written to the shared memory of the streaming end, it includes three stages: lock creation, lock acquisition, and lock release. Lock creation involves creating a corresponding mutex lock, hPStreamShareMutex, using CreateMutex, initially in an unowned state. Lock acquisition is performed by calling WaitForSingleObject(hPStreamShareMutex, INFINITE) before writing to the shared memory of the streaming end, waiting to acquire the lock (if the lock is already held, it blocks until the lock is released), ensuring that only one thread writes video frame data at a time. Lock release is performed by calling ReleaseMutex(hPStreamShareMutex) after the video frame data is written, releasing the lock and allowing other threads to access it.

[0100] When the second target video frame is written to the shared memory of the virtual camera, it includes three stages: lock creation, lock acquisition, and lock release. Lock creation involves creating a corresponding mutex lock, hVirCamShareMutex, using CreateMutex, initially in an unowned state. Lock acquisition involves calling WaitForSingleObject(hVirCamShareMutex, INFINITE) before writing to the shared memory of the virtual camera, waiting to acquire the lock (if the lock is already held, it blocks until the lock is released), ensuring that only one thread writes video frame data at a time. Lock release occurs after the video frame data is written, calling ReleaseMutex(hVirCamShareMutex) to release the lock, allowing other threads to access it.

[0101] Additionally, when reading its shared memory at the streaming end, the streaming process opens its shared memory through OpenFileMapping (specifying the mapping name "LPS_Stream_Share_Mem"), obtains the memory pointer through MapViewOfFile, and directly reads YUV420P format data without any additional processing. After reading is complete, UnmapViewOfFile is called to release the mapping (without closing the shared memory, ensuring that subsequent frames can continue to be written).

[0102] When the virtual camera reads its shared memory, the virtual camera driver opens its shared memory through OpenFileMapping (specifying the mapping name "VirCam_Share_Mem"), obtains the memory pointer through MapViewOfFile, reads landscape YUV420P data, and uses it as the output frame of the virtual camera for front-end preview applications (such as browsers and video conferencing software) to call.

[0103] Based on the above description, the video frame sharing output processing method provided in this application has the following advantages:

[0104] 1) Reduced resource consumption. Because video frames are processed only once, compared with the existing dual-process independent processing solution, the CPU utilization rate is reduced by 40%-50% (in actual testing at 1080p@30fps, the CPU utilization rate of this solution is 22%, while that of the existing solution is 45%), and the GPU utilization rate is reduced by 30%-40%, thus solving the problem of resource waste caused by repeated processing.

[0105] 2) Consistent screen synchronization. Through shared memory and synchronization lock mechanism, the streaming end and the virtual camera read the same batch of processed video frames, and the screen latency is controlled within 10ms (far below the 50ms threshold that the human eye can perceive). In addition, the horizontal and vertical screen adaptation logic is standardized to avoid layout offset and solve the problems of synchronization latency and consistency.

[0106] 3) Flexible compatibility with both landscape and portrait orientations. The strategy of direct scaling in landscape mode and precise black-and-white space in portrait mode supports both 16:9 landscape and 9:16 portrait formats captured by the camera, and the virtual camera output is always in standard 16:9 landscape mode, adapting to front-end preview requirements and solving the problem of format adaptation limitations.

[0107] 4) Strong cross-process versatility. Implemented based on the Windows shared memory API, it supports streaming clients and virtual camera modules with different development languages ​​(C++, C#, etc.) and different architectures (32-bit / 64-bit). No modification to the core logic of existing modules is required; only a shared memory read interface needs to be added, reducing integration costs.

[0108] The above describes a video frame sharing output processing method provided by the embodiments of this application. The following will describe the apparatus for performing the above video frame sharing output processing method.

[0109] See Figure 5 , Figure 5 This is a schematic diagram of a video frame sharing output processing device provided in an embodiment of this application. Figure 5 As shown in the figure, an embodiment of this application provides a video frame sharing output processing apparatus, comprising:

[0110] The frame acquisition module 501 is used to acquire the raw video frames captured by the camera;

[0111] The frame conversion module 502 is used to convert the original video frame into a first target video frame according to the format configuration of the streaming end, and to convert the original video frame into a second target video frame according to the format configuration of the virtual camera. The streaming end and the virtual camera have their own shared memory pre-created.

[0112] The frame writing module 503 is used to write the first target video frame to the shared memory of the streaming end, and to write the second target video frame to the shared memory of the virtual camera.

[0113] In one possible implementation, the frame conversion module 502, used to convert the original video frame into a first target video frame according to the format configuration of the streaming end, is specifically used for:

[0114] Obtain the original resolution of the original video frame and the first target resolution in the format configuration of the streaming end; if the original resolution and the first target resolution are the same, use the original video frame as the first target video frame; if the original resolution and the first target resolution are different, scale the original video frame with the original resolution as the initial value and the first target resolution as the target value to obtain the first target video frame.

[0115] In one possible implementation, the frame conversion module 502, used to convert the original video frames into second target video frames according to the format configuration of the virtual camera, is specifically used for:

[0116] Obtain the original resolution of the original video frame and the second target resolution in the virtual camera's format configuration. The original resolution includes the original height and original width of the original video frame, and the second target resolution includes the target height and target width of the virtual camera. If the original height of the original video frame is greater than its original width, determine the original aspect ratio of the original video frame based on its original height and original width. Use the target height of the virtual camera as the target height of the original video frame, and determine the target width of the original video frame based on its target height and original aspect ratio. Scale the original video frame using its original height and original width as initial values ​​and its target height and target width as target values ​​to obtain the second target video frame. Determine the black space width of the second target video frame based on its target width and the target width of the virtual camera.

[0117] In one possible implementation, the frame writing module 503 for writing the second target video frame to the shared memory of the virtual camera is specifically used for:

[0118] Adjust the output component pointer of the shared memory of the virtual camera according to the black space width, and perform black space filling.

[0119] In one possible implementation, the frame conversion module 502, used to convert the original video frames into second target video frames according to the format configuration of the virtual camera, is further used for:

[0120] If the original height of the original video frame is less than its original width, compare the original height of the original video frame with the target height of the virtual camera, and compare the original width of the original video frame with the target width of the virtual camera. If the original height and the target height of the original video frame are the same as the target height of the virtual camera, and the original width and the target width of the original video frame are the same as the target width of the virtual camera, then the original video frame is used as the second target video frame. If the original height and the target height of the original video frame are different from the target height of the virtual camera, or the original width and the target width of the original video frame are different from the target width of the virtual camera, then the original video frame is scaled using the original height and the original width of the original video frame as initial values ​​and the target height and the target width of the virtual camera as target values ​​to obtain the second target video frame.

[0121] In one possible implementation, the frame writing module 503 is specifically used for:

[0122] The first target video frame is written to the shared memory of the streaming end based on the mutex lock mechanism, and the second target video frame is written to the shared memory of the virtual camera.

[0123] It should be noted that the detailed functions of each module in the embodiments of this application can be found in the corresponding disclosure of the above-mentioned video frame sharing output processing method embodiments, and will not be repeated here.

[0124] This application also provides an electronic device in its embodiments. See also... Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0125] like Figure 6 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0126] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0127] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the video frame sharing output processing methods provided in this application.

[0128] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the video frame sharing output processing methods provided in this application.

[0129] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0132] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method of processing video frame sharing output, characterized by, The method for processing a video frame shared output comprises: acquiring an original video frame collected by a camera; converting the original video frame into a first target video frame according to a format configuration of a push stream end, and converting the original video frame into a second target video frame according to a format configuration of a virtual camera, the push stream end and the virtual camera being pre-created with respective shared memories; writing the first target video frame into the shared memory of the push stream end, and writing the second target video frame into the shared memory of the virtual camera.

2. The method of Claim 1, wherein, The method for converting the original video frame into the first target video frame according to the format configuration of the push stream end comprises: acquiring an original resolution of the original video frame and a first target resolution in the format configuration of the push stream end; in a case where the original resolution is the same as the first target resolution, taking the original video frame as the first target video frame; in a case where the original resolution is different from the first target resolution, scaling the original video frame to obtain the first target video frame, taking the original resolution as an initial value and taking the first target resolution as a target value.

3. The method of Claim 1, wherein, The method for converting the original video frame into the second target video frame according to the format configuration of the virtual camera comprises: acquiring an original resolution of the original video frame and a second target resolution in the format configuration of the virtual camera, the original resolution comprising an original height and an original width of the original video frame, the second target resolution comprising a target height and a target width of the virtual camera; in a case where the original height of the original video frame is greater than the original width thereof, determining a vertical screen original ratio of the original video frame according to the original height and the original width of the original video frame; taking the target height of the virtual camera as a target height of the original video frame, and determining a target width of the original video frame according to the target height of the original video frame and the vertical screen original ratio; scaling the original video frame to obtain the second target video frame, taking the original height and the original width of the original video frame as initial values and taking the target height and the target width of the original video frame as target values; determining a black leaving width of the second target video frame based on the target width of the original video frame and the target width of the virtual camera.

4. The method of claim 3, wherein, The method for writing the second target video frame into the shared memory of the virtual camera comprises: adjusting an output component pointer of the shared memory of the virtual camera according to the black leaving width, and performing black leaving filling.

5. The method of claim 3, wherein, The method for converting the original video frame into the second target video frame according to the format configuration of the virtual camera further comprises: in a case where the original height of the original video frame is less than the original width thereof, comparing the original height of the original video frame with the target height of the virtual camera, and comparing the original width of the original video frame with the target width of the virtual camera; if the original height of the original video frame is the same as the target height of the virtual camera and the original width of the original video frame is the same as the target width of the virtual camera, the original video frame is taken as the second target video frame; if the original height of the original video frame is different from the target height of the virtual camera or the original width of the original video frame is different from the target width of the virtual camera, the original video frame is scaled with the original height and the original width of the original video frame as initial values and the target height and the target width of the virtual camera as target values to obtain the second target video frame.

6. The method of Claim 1, wherein, the writing of the first target video frame into the shared memory of the push stream end and the writing of the second target video frame into the shared memory of the virtual camera comprises: the writing of the first target video frame into the shared memory of the push stream end and the writing of the second target video frame into the shared memory of the virtual camera based on a mutual exclusion lock mechanism.

7. A processing device for video frame sharing output, characterized in that, the processing apparatus for video frame shared output comprises: a frame acquisition module configured to acquire an original video frame collected by a camera; a frame conversion module configured to convert the original video frame into a first target video frame according to a format configuration of a push stream end and to convert the original video frame into a second target video frame according to a format configuration of a virtual camera, the push stream end and the virtual camera being pre-created with respective shared memories; a frame writing module configured to write the first target video frame into the shared memory of the push stream end and to write the second target video frame into the shared memory of the virtual camera.

8. A computer program product, characterised in that, comprising computer readable instructions, when the computer readable instructions are run on an electronic device, the electronic device is caused to implement the processing method for video frame shared output as claimed in any one of claims 1 to 6.

9. An electronic device, comprising: comprising at least one processor and a memory connected with the processor, wherein: the memory is configured to store a computer program; the processor is configured to execute the computer program to enable the electronic device to implement the processing method for video frame shared output as claimed in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device is enabled to implement the processing method for video frame shared output as claimed in any one of claims 1 to 6.

Images