Methods, devices, equipment and storage media for recording instructional videos

By acquiring screen and captured image frames during teaching video recording and performing human image segmentation, the problems of camera reflection interference and green screen limitations are solved, enabling the synthesis of clear courseware content with real classroom scenes, and is suitable for electronic devices with various hardware configurations.

CN123123525APending Publication Date: 2026-11-17SHENZHEN HONGHE INNOVATION INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611416431.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing teaching video recording methods, the camera is easily affected by screen reflections, resulting in blurry courseware content. Furthermore, green screen keying requires a special venue, which is not suitable for deployment in ordinary classrooms, and cannot achieve a clear and natural synthesis of courseware content and teacher image.

Method used

By acquiring screen image frames from large-screen devices and image frames captured by camera devices, the execution provider component performs portrait segmentation processing. Combined with the compositing processing of multiple moments, an instructional video is generated, avoiding the need for green screen scenes and adapting to electronic devices with different hardware configurations.

Benefits of technology

In ordinary classrooms, the courseware content is clear and glare-free, effectively restoring the real classroom scene and improving the flexibility and scene adaptability of video generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123525A_ABST
    Figure CN123123525A_ABST
Patent Text Reader

Abstract

This application relates to the field of teaching video processing technology, and provides a teaching video recording method, apparatus, device, and storage medium, including: acquiring screen image frames captured from a large-screen device and captured image frames captured from a shooting device at the same time, wherein the screen image frames are images representing the content of teaching courseware captured on the screen, and the captured image frames are images containing people captured by the shooting device; determining a portrait segmentation image based on the captured image frames and at least one execution provider component, wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frames; and performing composite processing on the screen image frames and portrait segmentation images corresponding to several moments to generate a teaching video. Through portrait segmentation and image synthesis, green screen deployment is unnecessary, screen reflection problems are solved, the flexibility of video generation and video clarity are improved, and the viewing experience of teaching videos is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of teaching video processing, and in particular relates to a teaching video recording method, apparatus, device and storage medium. Background Technology

[0002] With the development of online teaching, the application of teaching videos is becoming increasingly widespread. Teaching videos have high requirements for the clarity of courseware and the realism of classroom scenes.

[0003] The teaching video recording method in related technologies involves filming and recording the teacher's lecture in front of a large screen in the classroom. However, there is a problem that the video recording is easily affected by screen reflections, resulting in blurry courseware content.

[0004] Therefore, how to generate clear and authentic teaching videos is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a teaching video recording method, apparatus, device, and storage medium, which can improve video clarity and restore the visual experience of a real classroom.

[0006] In a first aspect, embodiments of this application provide a method for recording instructional videos, including: The system acquires screen image frames captured from a large-screen device and captured image frames captured from a camera device at the same time. The screen image frames are images representing the content of teaching courseware captured on the screen, and the captured image frames are images containing people captured by the camera device. Based on the captured image frame, a portrait segmentation image is determined based on at least one execution provider component, the execution provider component being used to specify hardware for performing image segmentation processing, the portrait segmentation image representing an image of a person segmented from the captured image frame; The screen image frames and portrait segmentation images corresponding to several of the aforementioned times are synthesized to generate an instructional video.

[0007] Secondly, embodiments of this application provide a teaching video recording device, including: The image acquisition module acquires screen image frames captured from the large-screen device and captured image frames captured from the shooting device at the same time. The screen image frames are images representing the content of the teaching courseware captured on the screen, and the captured image frames are images containing people captured by the shooting device. A portrait segmentation module is used to determine a portrait segmentation image based on the captured image frame and at least one execution provider, wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frame; The video generation module is used to synthesize screen image frames and human portrait segmentation images corresponding to several times to generate teaching videos.

[0008] Thirdly, embodiments of this application provide a large-screen device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method described in the first aspect above.

[0011] In this embodiment, screen image frames and captured image frames can be acquired separately at various moments during the teacher's lecture, resulting in images containing clear teaching materials and images containing people. The large-screen device can be equipped with at least one execution provider component. This component determines the hardware required for image segmentation processing, thereby performing image segmentation on the captured image frames to obtain segmented images. The screen image frames and segmented images are then synthesized, and the combined results from multiple moments are used to obtain the teaching video. Image segmentation eliminates the need for a green screen, reducing scene requirements and increasing the flexibility of video generation. By acquiring screen image frames and captured image frames separately, the teaching materials in the video are clear and glare-free, effectively reproducing a realistic classroom scene. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a system architecture diagram provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the teaching video recording method provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the teaching video recording method provided in the embodiments of this application; Figure 4This is a flowchart of the video synthesis process provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the teaching video recording method provided in the embodiments of this application; Figure 6 This is a flowchart illustrating the teaching video recording method provided in the embodiments of this application; Figure 7 This is a flowchart illustrating the teaching video recording method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the teaching video recording device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0015] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0016] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0018] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0020] The teaching video recording method provided in this application can be applied to terminal devices such as mobile phones, tablets, all-in-one teaching machines, laptops, large-screen devices, netbooks, and smart blackboards. This application does not impose any restrictions on the specific type of terminal device.

[0021] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: DXGI: DirectX Graphics Infrastructure, the Windows graphics infrastructure interface, is used for high-performance screen capture.

[0022] CUDA: Compute Unified Device Architecture.

[0023] DirectML: Direct Machine Learning.

[0024] CPU: Central Processing Unit.

[0025] EP: Execution Provider.

[0026] BGR: Blue-Green-Red, is a three-channel image pixel storage format.

[0027] BGRA: Blue-Green-Red-Alpha, is a four-channel image pixel storage format.

[0028] Most current teaching videos are recorded using methods such as direct camera shooting, screen recording with picture-in-picture, and green screen keying. Direct camera shooting allows the camera to simultaneously capture both the teacher and the large screen, but light reflection (moiré, glare) causes the courseware content on the large screen to become blurry. Screen recording with picture-in-picture allows software to record the screen while simultaneously overlaying a small camera window in a corner, but the drawback is that the teacher's image is separated from the courseware image, failing to simulate a realistic classroom atmosphere. Green screen keying involves the teacher standing in front of a green screen, with the teacher's image extracted using chroma keying and overlaid onto the courseware; however, it requires a dedicated green screen area, and the green screen can obscure the large screen image, making it unsuitable for deployment in ordinary classrooms.

[0029] The embodiments of this application can be applied to ordinary classroom scenarios without green screens, achieving two goals simultaneously: clear courseware content and natural integration of the teacher and courseware. This provides both a realistic classroom atmosphere and clear courseware visuals.

[0030] The system architecture used in the embodiments of this application Figure 1 This is a system architecture diagram of the teaching video recording method according to an embodiment of this application. It includes a terminal 140, an Internet 130, a gateway 120, a server 110, etc.

[0031] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0032] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0033] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0034] General Description of Embodiments in this Application According to one embodiment of this application, a method for recording teaching videos is provided. The teacher delivers a lesson in front of a large screen. The system captures the teacher's image using a camera or other imaging device, and uses DXGI to capture the large screen view. The system then segments and composites the teacher's image onto the courseware screen in real time, ultimately recording a teaching video overlaid on the courseware. The entire recording process requires no green screen and can be deployed in any ordinary classroom.

[0035] Figure 2 A flowchart illustrating the method for recording instructional videos, such as... Figure 2 As shown, a teaching video recording method according to one embodiment of this application may include: Step 210: Acquire screen image frames captured from the large screen device and captured image frames captured from the camera device at the same time; wherein, the screen image frame is an image representing the content of the teaching courseware captured on the screen, and the captured image frame is an image containing people captured by the camera device. Step 220: Based on the captured image frame, determine a portrait segmentation image using at least one execution provider; wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frame; Step 230: Combine the screen image frames and portrait segmentation images corresponding to several time points to generate a teaching video.

[0036] Steps 210-230 are described in detail below.

[0037] In step 210, this embodiment applies to an electronic device, which can be a large-screen device, i.e., a device equipped with a large display screen. The large-screen device can be a teaching host used for recording lectures in a typical classroom environment. The large-screen device can run programs such as screen capture, camera capture, image segmentation, and video compositing. During the lecture, courseware is displayed on the screen of the large-screen device, and the teacher stands in front of the courseware to lecture. For each moment, a corresponding screen image frame and a captured image frame can be acquired. The screen image frame is an image containing clear courseware content obtained by cropping the screen, and the captured image frame is an image containing the teacher, captured by a camera. The camera can be a camera, webcam, etc. For example, a camera can be pointed directly at the podium, and the captured image frame can include the teacher or the screen.

[0038] Screen image frames can be captured via the DirectX Graphics Infrastructure (DXGI) interface. DXGI's desktop copy interface is used for high-performance screen capture, directly reading the content output by the graphics processor. This avoids glare, moiré patterns, and reflections caused by optical screen capture, ensuring clear and complete courseware content. Screen image frames can be acquired by a dedicated screen capture thread. In case of capture errors, the data from the previous frame can be reused to maintain a stable frame rate in subsequent compositing processes.

[0039] The captured image frame is an image containing a person, acquired by the capturing device. This embodiment does not specifically limit the type of capturing device. The capturing device can be a universal Serial Bus (USB) camera, acquiring the captured image frame using a video reading class from an open-source computer vision library. The captured image frame can be acquired and output by a separate camera acquisition thread. This embodiment does not specifically limit the position of the capturing device or the shooting angle.

[0040] In this embodiment, the acquisition period of screen image frames and captured image frames can be the same, that is, each screen image frame can correspond to a captured image frame.

[0041] In step 220, the electronic device is configured with at least one Execution Provider (EP). The EP is a hardware adaptation component within the ONNX Runtime inference engine and serves as the execution backend of the ONNX Runtime. It specifies the hardware unit responsible for image segmentation processing, i.e., determining which hardware performs the inference computation. ONNX Runtime is a deep learning inference runtime library. An EP can be selected from multiple options. Based on captured image frames, the hardware corresponding to that EP performs portrait segmentation processing to obtain a segmented portrait image. For example, EPs can be selected and adapted according to a preset priority. When a high-priority EP becomes unavailable, the system can automatically switch to the next available EP, thus adapting to electronic devices with different hardware configurations. For instance, the EP can be an execution provider based on the Microsoft Direct Machine Learning Interface or a central processing unit (CPU) execution provider.

[0042] A portrait segmentation image represents a person-related image segmented from a captured image frame. The size of the portrait segmentation image and the captured image frame can be the same, and the background of the person in the captured image frame can be hidden in the portrait segmentation image. For example, the portrait segmentation image can be a grayscale image in the form of an alpha mask, where the pixel values ​​of different pixels represent the transparency information at the corresponding locations. The portrait segmentation image can be obtained through inference output from a lightweight deep learning model. Gaussian blur processing can also be applied to the portrait segmentation image to achieve a smooth transition of the portrait contour edges, improving the accuracy of subsequent video synthesis.

[0043] In step 230, the screen image frames and portrait segmentation images at each time point are continuously determined. Based on the corresponding screen image frames and portrait segmentation images at each time point, a teaching video is generated. During the generation of the teaching video, a homography matrix can be used to perform a perspective transformation of the portrait image from the camera coordinate system to the screen coordinate system, ensuring that the position of the portrait in the composite image matches the actual position of the portrait relative to the large screen in the physical world. Alpha blending can also be performed by combining the transparency information provided by the portrait segmentation image, achieving a natural fusion of the portrait and the courseware content. In this embodiment, the process of performing alpha blending is not specifically limited. Blending can be optimized using a parallel traversal loop interface (parallel_for_) for single-pass optimization, fully utilizing the multi-core performance of the Central Processing Unit (CPU).

[0044] For each moment, the corresponding screen image frame and the segmented human image can be blended to obtain a composite image frame for that moment. The composite image frame includes both the person and clear courseware content. After image blending, the resulting composite image frame can be played directly as a video, or it can be sent to an encoder for compression encoding to obtain a video stream. The compressed video stream can then be saved to the local storage medium of the electronic device to obtain the final teaching video.

[0045] In this embodiment, by acquiring screen image frames captured by the screen and image frames captured by the shooting device, a human figure segmentation image is obtained with the help of an execution provider with adaptive hardware. Then, a teaching video is generated based on the screen image frames and the human figure segmentation image. The teaching video can be recorded in a normal classroom environment without the need for special facilities such as green screens. This ensures that the courseware content is clear and that the human figure and courseware image are naturally composited. At the same time, it can be deployed and used on a variety of electronic devices with different hardware configurations.

[0046] In one embodiment, the method further includes: preprocessing the captured image frame to obtain a preprocessed captured image frame; wherein the preprocessing includes scaling and horizontal mirroring; and determining a portrait segmentation image based on the preprocessed captured image frame and at least one execution provider in the electronic device.

[0047] Specifically, preprocessing is performed on the captured image frames to obtain preprocessed image frames. Preprocessing operations include scaling and horizontal mirroring. The captured image frames originate from image data output by the camera acquisition thread, and the preprocessing operations can be completed before the images are sent to the portrait segmentation process. Scaling can adapt to preset size requirements, reducing the amount of data computation in subsequent portrait segmentation processes; horizontal mirroring can correct the left-right inversion problem caused by camera imaging, ensuring that the perspective of the person in the output image conforms to the visual habits of direct human observation. The preprocessed image frames can then be processed in step 220, i.e., used by the portrait segmentation process.

[0048] For example, scaling can involve downsampling or upsampling the captured image frames. Scaling can be implemented using image interpolation methods such as nearest neighbor interpolation or bilinear interpolation. Horizontal mirroring can be performed by calling the image transformation interface within an open-source computer vision library. For instance, the MJPEG format can be negotiated to reduce bandwidth pressure, and captured image frames can be read repeatedly and horizontal mirroring can be performed.

[0049] After obtaining the preprocessed captured image frames, the portrait segmentation image is determined based on at least one EP configured within the electronic device. That is, the image after scaling and mirroring is used as the input for portrait segmentation, replacing the original unprocessed captured image frames.

[0050] In this embodiment, preprocessed image frames are obtained by scaling and horizontal mirroring the captured image frames. The preprocessed image frames are then used to complete portrait segmentation. This can adapt to the input specifications of the portrait segmentation model, reduce the computational overhead of the inference process, and correct the mirror deviation of the camera imaging, thereby improving the visual quality of the final synthesized output teaching video.

[0051] In one embodiment, the method further includes: if a preset hardware encoder is deployed in the electronic device, the teaching video is compressed using the preset hardware encoder, and the compressed teaching video is stored; if the electronic device does not have a preset hardware encoder, the teaching video is compressed using a pre-packaged software encoder, and the compressed teaching video is stored.

[0052] Specifically, the teaching video is a sequence of composite image frames obtained by fusing human images with courseware images. Compression processing is used to reduce the data size of the teaching video, making it easier to permanently save the teaching video to the local storage medium of electronic devices and avoid the original uncompressed image frames occupying too much storage space.

[0053] Electronic devices can deploy both hardware encoders and software encoders. A hardware encoder is a dedicated encoding hardware unit integrated within the graphics processing unit (GPU) chip. It can perform encoding work independently of the GPU's general-purpose computing core and does not consume the GPU's general-purpose computing resources. A software encoder is an encoding module encapsulated based on the operating system's multimedia components. It can complete video compression tasks when the hardware encoder is unavailable. For example, the hardware encoder could be NVENC H.264, and the software encoder could be h264_mf.

[0054] Based on the actual configuration of the electronic device's hardware resources, a suitable encoder can be selected to compress and store the teaching videos. Encoder access can be protected using mutex locks to mitigate the risk of program errors caused by concurrent multi-threaded access. For example, the NVENC H.264 hardware encoder is used preferentially, and if it becomes unavailable, it is rolled back to h264_mf. Recording and closing operations are protected using std::mutex mutex locks to prevent multi-threaded race conditions and crashes. std::mutex is a mutex lock component. Both hardware and software encoders can output video streams conforming to common video encoding standards. The encoded video streams can be written to the electronic device's local storage media to generate playable video files.

[0055] In this embodiment, by determining whether the electronic device is equipped with a preset hardware encoder, the corresponding hardware encoder or pre-packaged software encoder is selected to complete the compression and storage of the teaching video. This can give priority to the performance advantages of dedicated encoding hardware, reduce the occupation of the electronic device's computing resources during the encoding process, and at the same time, rely on the software encoder to ensure that the video recording and storage process can be completed normally when the corresponding hardware conditions are lacking, thereby improving the adaptability of electronic devices with different hardware configurations.

[0056] In this embodiment, screen image frames and captured image frames can be acquired separately at various moments during the teacher's lecture, resulting in images containing clear courseware content and images containing people. The electronic device can be equipped with at least one execution provider component. This component determines the hardware required for image segmentation processing, thereby performing image segmentation on the captured image frames to obtain segmented images. The screen image frames and segmented images are then synthesized, and the results from multiple moments are combined to obtain a teaching video. Image segmentation eliminates the need for green screen scenes, reducing scene requirements and increasing the flexibility of video generation. By acquiring screen image frames and captured image frames separately, the courseware content in the teaching video is clear and glare-free, effectively reproducing a realistic classroom scene.

[0057] Figure 3 A flowchart illustrating the method for recording instructional videos, such as... Figure 3 As shown, a teaching video recording method according to one embodiment of this application may include: Step 310: Acquire screen image frames captured from the large screen device and captured image frames captured from the camera device at the same time; wherein, the screen image frame is an image representing the content of the courseware captured on the screen, and the captured image frame is an image containing people captured by the camera; Step 320: Create a model inference session based on at least one execution provider in the electronic device; wherein the model inference session represents an instance object of the model after it is running in memory; Step 330: Run the preset deep learning model based on the model inference session to obtain the portrait segmentation image corresponding to the captured image frame; wherein, the preset deep learning model is used for portrait segmentation; Step 340: Combine the screen image frames and portrait segmentation images corresponding to several time points to generate a teaching video.

[0058] Steps 310 and 340 have been described in detail above and will not be repeated here. Steps 320 and 330 will be described in detail below.

[0059] In step 320, the model inference session is an instance object formed after the artificial intelligence (AI) model is loaded into memory. It is responsible for scheduling the EP, managing model weights, receiving input image data, and outputting inference results.

[0060] Based on various EPs (Engineers) in the electronic device, the final EP to be used is determined, and a model inference session is created based on the finally selected EP. That is, the model inference session creation process binds to the corresponding EP, allowing subsequent AI model computation tasks to be handled by the hardware unit specified by the EP. For example, when creating a model inference session, a higher-priority EP can be tried first. If the corresponding EP cannot complete session creation normally, another available EP can be used to continue trying session creation. In this embodiment, the session creation process is not specifically limited.

[0061] In step 330, a model inference session is first created based on at least one available EP within the electronic device, and then a preset AI model is run based on the created model inference session. In this embodiment, the AI ​​model is a lightweight deep learning model, for example, the MODNet deep learning model, which is a lightweight deep learning model for human face segmentation.

[0062] The captured image frame or a pre-processed captured image frame is used as input to the model to obtain a portrait segmentation image corresponding to the portrait image frame, thus completing the portrait segmentation process. The preset deep learning model can be a lightweight portrait segmentation deep learning model, which outputs a portrait segmentation image representing the transparency information of the person. This embodiment does not impose specific limitations on the model architecture.

[0063] In other words, after creating the model inference session, a pre-defined deep learning model can be run based on this session. This pre-defined deep learning model is specifically designed for portrait segmentation tasks and can distinguish between the person region and the background region from a captured image frame containing a person, outputting a portrait segmentation image. The output portrait segmentation image can be further processed for edge smoothing before being sent to a cache queue for use in subsequent teaching video generation. For example, during the operation of the pre-defined deep learning model, the input captured image frame can be adapted to meet the model's computational requirements. The portrait segmentation image can be a grayscale image, and the pixel values ​​within the image can represent the transparency of the corresponding pixel location belonging to the person region.

[0064] In this embodiment, a model inference session is created based on at least one execution provider, and then a preset deep learning model for portrait segmentation is run using the model inference session to obtain a portrait segmentation image. The deep learning model can be bound to execution providers corresponding to different hardware, making full use of the available hardware resources of electronic devices to complete the portrait segmentation operation, and ensuring that the portrait segmentation process can be executed normally on electronic devices with different configurations.

[0065] In one embodiment, the execution provider in the electronic device includes at least one of a CUDA execution provider, a DirectML execution provider, and a CPU execution provider; creating a model inference session based on at least one execution provider in the electronic device includes: initializing the CUDA execution provider; if the CUDA execution provider initialization is successful, creating a model inference session through the CUDA execution provider; if the CUDA execution provider initialization fails, initializing the DirectML execution provider; if the DirectML execution provider initialization is successful, creating a model inference session through the DirectML execution provider; if the DirectML execution provider initialization fails, initializing the CPU execution provider, creating a model inference session through the CPU execution provider.

[0066] Specifically, an EP (Execution Processor) is a hardware adaptation component within an AI inference engine, used to schedule operators of a deep learning model to corresponding hardware units for computation. Electronic devices can be configured with various types of EPs, such as CUDA EPs, DirectML EPs, and CPU EPs. A CUDA EP is a CUDA-based execution backend that utilizes the CUDA cores of a GPU for inference acceleration; a DirectML EP is a Microsoft DirectML-based execution backend that utilizes a graphics processing unit (GPU) supporting the DirectX 12 (DX12) graphics application programming interface for inference; and a CPU EP is a CPU execution backend, serving as a backup solution when no GPU is available, accelerating performance through multi-threaded parallelism.

[0067] Initialization checks are performed on various types of EPs in a pre-defined order. If the initialization of a high-priority EP is completed normally, that EP is used to create the model inference session. If the initialization of the current EP fails, the initialization process is performed on the next type of EP in sequence until a working EP is found and the model inference session is created.

[0068] The initialization process of EP can include hardware capability detection, driver version verification, and inference runtime environment configuration. EP initialization failure can be caused by various factors such as missing hardware, driver incompatibility, and abnormal resource usage. After the model inference session is created, the preset deep learning model can be loaded to perform human image segmentation calculation.

[0069] For example, the AI ​​inference engine automatically detects the hardware environment and selects the optimal EP (Extended Processing Unit) based on priority from CUDA to DirectML to CPU. If it fails, it automatically downgrades, allowing the same program to be deployed in classrooms with different hardware configurations, achieving multi-EP adaptive inference. In other words, it first attempts to load CUDA EP functions; if available, it uses the CUDA EP to create a model inference session. If unavailable, it tries the DirectML EP; if both fail, it uses the CPU EP. If the CPU EP is also unavailable, video compositing fails, and a prompt message can be sent to the user, reminding them to check the configuration.

[0070] After the initialization of the corresponding EP and the creation of the model inference session are completed sequentially, the generated model inference session will be bound to the currently available EP. When running the preset deep learning model, the inference task will be undertaken by the hardware unit corresponding to that EP. For example, the model inference session created using a CUDA EP can fully utilize the parallel computing capabilities of the graphics processor, thereby improving the running efficiency of portrait segmentation processing; the model inference session created using a CPU EP can ensure that the portrait segmentation process is not interrupted even when all graphics processor acceleration conditions are not met. The model inference session obtained through the above priority selection can receive preprocessed captured image frames as input and output portrait segmentation images, which can be used in the subsequent teaching video generation stage.

[0071] In this embodiment, by initializing CUDA EP, DirectML EP, and CPU EP in sequence according to preset priorities, and selecting the successfully initialized EP to create a model inference session, the system can automatically adapt to the actual hardware conditions of electronic devices, and can normally complete the loading of deep learning models and human image segmentation operations on electronic devices with different hardware configurations.

[0072] In one implementation, a preset deep learning model is run based on a model inference session to obtain a portrait segmentation image corresponding to a captured image frame, including: running a preset deep learning model based on a model inference session to obtain an initial segmentation image corresponding to a captured image frame; wherein the initial segmentation image is a single-channel grayscale image with the same size as the captured image frame; and performing Gaussian blur processing on the initial segmentation image corresponding to the captured image frame to obtain the portrait segmentation image corresponding to the captured image frame.

[0073] Specifically, the initial segmented image is directly output by a pre-defined deep learning model after model inference and computation. It is a single-channel grayscale image with the same frame size as the captured image. The grayscale value of each pixel in the initial segmented image can mark whether the corresponding location in the image belongs to the human or background region. The initial segmented image is an alpha mask, where the mask represents the transparency of each pixel, used for smooth compositing of the human image and background. For example, 0 represents complete transparency, and 255 represents complete opacity.

[0074] The initial segmentation image is the raw inference output without edge smoothing. This initial segmentation image can be processed to obtain the final portrait segmentation image. For example, Gaussian blurring can be applied to the initial segmentation image to smooth the edges of the portrait contour, resulting in the final portrait segmentation image used for image compositing. In the initial segmentation image, grayscale values ​​represent the transparency of pixel locations. Pixels near the edges of the portrait contour can generate a gradual transition effect after Gaussian blurring. Gaussian blurring can be performed using the corresponding image processing interface within an open-source computer vision library. The intensity of the Gaussian blurring can be adjusted according to the actual image effect requirements. In this embodiment, the specific process of Gaussian blurring is not limited. For example, different forms of convolution kernels can be used to perform image smoothing operations. Furthermore, Gaussian blurring can be performed on the entire initial segmentation image or only on the area surrounding the portrait contour in the initial segmentation image for local smoothing.

[0075] After obtaining the portrait segmentation image through Gaussian blur processing, the portrait segmentation image can be temporarily stored in the corresponding result queue for easy retrieval and retrieval in the compositing process. The transparency information saved in the portrait segmentation image can be used in the image fusion stage to complete the alpha blending of the portrait image and the screen image frame, reducing the jagged edges and harsh contours of the portrait in the composite image.

[0076] In this embodiment, an initial segmentation image in single-channel grayscale form is obtained by running a preset deep learning model. Then, Gaussian blur processing is performed on the initial segmentation image to obtain a portrait segmentation image. This can make the contour edges of the portrait area form a natural gradient transition, eliminate the harsh jaggedness brought by the directly output segmentation image, and improve the visual effect after the portrait image and the courseware screen are merged.

[0077] Figure 4This is a flowchart of the video compositing process. First, the system is initialized, including DXGI, camera, AI model, and OpenGL (Open Graphics Library). OpenGL is used for GPU-accelerated image rendering and preview display. Based on the AI ​​inference engine, various EPs (Executable Processors) are adaptively selected, with the selection order being CUDA EP, DirectML EP, and CPU EP. Four parallel threads are started: a camera acquisition thread, a screen acquisition thread, an AI segmentation thread, and a composite image thread. The camera acquisition thread acquires captured image frames, the screen acquisition thread acquires screen image frames, the AI ​​segmentation thread generates segmented human images, and the composite image thread generates composite image frames. After obtaining the sequence of composite image frames, encoding and recording can be performed simultaneously, along with file storage, while the images are displayed in a window for user preview.

[0078] In this embodiment, screen image frames and captured image frames can be acquired separately at various moments during the teacher's lecture, resulting in images containing clear courseware content and images containing people. The device can be equipped with at least one execution provider component. This component determines the hardware required for image segmentation processing, thereby performing image segmentation on the captured image frames to obtain segmented images. The screen image frames and segmented images are then synthesized, and the combined results from multiple moments are used to obtain a teaching video. Image segmentation eliminates the need for green screen scenes, reducing scene requirements and increasing the flexibility of video generation. By acquiring screen image frames and captured image frames separately, the courseware content in the teaching video is clear and glare-free, effectively reproducing a realistic classroom scene.

[0079] Figure 5 A flowchart illustrating the method for recording instructional videos, such as... Figure 5 As shown, a teaching video recording method according to one embodiment of this application may include: Step 510: Determine the current acquisition window based on the time and the preset time length; wherein, the current acquisition window represents the time window of the screen image frame used for acquisition. Step 520: If no screen image frame is captured in the current capture window, the previously captured screen image frame is reused as the screen image frame at that moment.

[0080] Step 530: Acquire the captured image frames at the same time, and determine the portrait segmentation image based on the captured image frames and at least one execution provider; wherein, the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents the image of the person segmented from the captured image frames; Step 540: Combine the screen image frames and portrait segmentation images corresponding to several time points to generate a teaching video.

[0081] Steps 530 and 540 have been described in detail above and will not be repeated here. Steps 510 and 520 will be described in detail below.

[0082] In step 510, for each moment, a current acquisition window can be determined. The current acquisition window is a time window used to acquire screen image frames at the corresponding moment. A time length can be preset, and the current acquisition window is determined based on the corresponding moment and the preset time length. For example, the current acquisition window can be a window with the current moment as the starting point and a preset time length. The preset time length can be a short time, for example, 5ms.

[0083] In step 520, the current acquisition window is used to limit the effective time range of the screen image frame acquisition operation. The screen image frame is the courseware content image obtained directly from the graphics processor side, which can avoid the reflection and moiré problems caused by optically capturing the screen.

[0084] For each moment, a corresponding acquisition time window is defined by combining that moment with a preset time length, serving as the current acquisition window. Within this current acquisition window, attempts are made to acquire screen image frames. If no new screen image frame is acquired by the end of the window, the screen image frame saved in the previous window can be directly used as the current screen image frame to avoid interrupting the overall compositing process due to screen acquisition timeout. For example, within the current acquisition window, the `AcquireNextFrame` interface can be called repeatedly to acquire screen image frames. `AcquireNextFrame` is a frame acquisition interface function provided by the DXGI desktop copy interface.

[0085] For example, screen image frames can be captured using the DirectX desktop copy interface. The capture process can run on a separate screen capture thread. The captured screen image frames can be cached in memory for subsequent reading and reuse. The preset time length can be adapted and adjusted according to the operating load of the electronic device.

[0086] In this embodiment, a short timeout detection and frame buffer reuse strategy is adopted. The current acquisition window is determined based on the current time and the preset time length. If a screen image frame cannot be acquired within the acquisition window, the previously acquired screen image frame is reused. This can avoid the problem of missing images caused by screen acquisition timeout, ensure the continuous execution of the image synthesis process, and maintain the frame rate stability of the teaching video output process.

[0087] In one implementation, acquiring screen image frames captured from a large-screen device at the same time includes: capturing and processing the images on the screen of the large-screen device through the DXGI desktop copy interface to obtain screen image frames at that time.

[0088] Specifically, the DXGI Desktop Duplication API (desktop duplication interface) can be used for hardware acceleration to capture large-screen images, thereby obtaining screen image frames. The DXGI Desktop Duplication API is a screen capture interface provided by the Windows operating system. It can directly read screen image resources output from the graphics processor's video memory, eliminating the need for optical imaging to acquire screen content and fundamentally avoiding glare, reflections, and moiré patterns caused by optical imaging. The DXGI Desktop Duplication API can run in a separate screen capture thread, providing screen image frames representing courseware content for the teaching video synthesis process.

[0089] For example, the Direct3D 11th Generation Graphics Application Programming Interface (D3D11) device and the DXGI Desktop Duplication API can be initialized to create staging textures. That is, the DXGI Desktop Duplication API can capture screen resources, which can be texture resources. These texture resources can be copied to staging textures, and then image format conversion can be performed. The converted image data can be stored in a memory cache as screen image frames for subsequent compositing processing.

[0090] This embodiment does not specifically limit the process of capturing screen image frames. For example, it can be: GPU memory desktop frame → Staging texture → Copy to CPU memory BGRA image → Use libyuv library for downsampling and scaling + BGRA to BGR conversion → Output BGR image as screen image frame. DXGI's output texture is a GPU-specific resource that the CPU cannot directly access; therefore, it needs to be copied via a Staging texture. The libyuv library is an open-source image conversion and processing tool library.

[0091] In this embodiment, the screen image is captured through the DirectX desktop copy interface to obtain screen image frames, avoiding the image distortion defects caused by optical acquisition, ensuring the clarity and completeness of the courseware content, and providing high-quality background images for subsequent teaching video synthesis.

[0092] In this embodiment, screen image frames and captured image frames can be acquired separately at various moments during the teacher's lecture, resulting in images containing clear courseware content and images containing people. The electronic device can be equipped with at least one execution provider component. This component determines the hardware required for image segmentation processing, thereby performing image segmentation on the captured image frames to obtain segmented images. The screen image frames and segmented images are then synthesized, and the results from multiple moments are combined to obtain a teaching video. Image segmentation eliminates the need for green screen scenes, reducing scene requirements and increasing the flexibility of video generation. By acquiring screen image frames and captured image frames separately, the courseware content in the teaching video is clear and glare-free, effectively reproducing a realistic classroom scene.

[0093] Figure 6 A flowchart illustrating the method for recording instructional videos, such as... Figure 6 As shown, a teaching video recording method according to one embodiment of this application may include: Step 610: Acquire screen image frames and determine the target acquisition backend from all video acquisition backends in the electronic device; wherein, the video acquisition backend is used to read images from the shooting device; Step 620: Obtain the captured image frames from the shooting device at the same time through the target acquisition backend; Step 630: Based on the captured image frame, determine a portrait segmentation image using at least one execution provider in the electronic device; wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frame; Step 640: Combine the screen image frames and portrait segmentation images corresponding to several time points to generate a teaching video.

[0094] Steps 630 and 640 have been described in detail above and will not be repeated here. Steps 610 and 620 will be described in detail below.

[0095] In step 610, the video acquisition backend is a functional module used to read image data from inside the camera. It can complete data interaction with the camera hardware and realize the reception and parsing of camera image data. Different video acquisition backends are adapted to different camera drivers and transmission protocols. Multiple video acquisition backends can be deployed inside the electronic device. For example, it can include a Microsoft Media Foundation (MSMF) acquisition backend and a DirectShow (DSHOW) multimedia framework acquisition backend.

[0096] MSMF (Multimedia Subsystem for Software) is a modern multimedia application programming interface for Windows, developed by Microsoft. In OpenCV, it serves as a camera video acquisition backend, connecting to USB camera hardware to read the video image data stream output by the camera and parsing the captured image frames. OpenCV is an open-source computer vision library that provides basic functions such as image acquisition, processing, and transformation. DSHOW (Digital Showdown) is an earlier Windows multimedia development framework from Microsoft, and it can also be used as an OpenCV camera video acquisition backend to access camera devices, receive and parse video streams, and has good compatibility with many older USB camera hardware devices.

[0097] One video acquisition backend can be selected from all video acquisition backends in the electronic device as the target acquisition backend. For example, the MSMF acquisition backend can be selected first. If the MSMF backend fails to open the camera device or the image reading is abnormal, another video acquisition backend can be switched to continue the acquisition work. In this embodiment, when the MSMF acquisition backend fails to initialize, the DSHOW acquisition backend can be enabled to complete the camera image acquisition work.

[0098] In other words, when one type of video acquisition backend fails to function properly, other available video acquisition backends can be used to continue image reading. The image data output by the video acquisition backend can be used as input for subsequent portrait segmentation processing, providing the raw image for generating the portrait segmentation image.

[0099] In step 620, the target acquisition backend is selected from all the video acquisition backends of the electronic device, and the camera data is read by relying on the selected target acquisition backend to obtain the captured image frame at the corresponding time, so as to ensure that the captured image frame can be stably output to the subsequent processing flow.

[0100] Once the target acquisition backend is identified, it can establish a data path with the camera, continuously receive the image data stream transmitted from the camera, and parse and output the captured image frames at the corresponding times. The acquired image frames can be sent to the camera acquisition queue for subsequent preprocessing stages. For example, during operation, the target acquisition backend can negotiate and determine the image encoding format output by the camera, thereby reducing bandwidth consumption during transmission from the camera to electronic devices. The captured image frames read by the target acquisition backend can also undergo preprocessing operations such as mirror flipping before being transmitted downstream.

[0101] In this embodiment, by determining the target acquisition backend from all video acquisition backends of the electronic device and using the target acquisition backend to read the captured image frames from the camera, image acquisition can be completed even when some video acquisition backends cannot be adapted to the camera hardware. This improves the device compatibility of the camera image acquisition process and ensures that the teaching video synthesis process can continuously acquire effective captured image frames.

[0102] In one embodiment, the video acquisition backend in the electronic device includes at least one of an MSMF acquisition backend and a DSHOW acquisition backend; determining the target acquisition backend from all video acquisition backends in the electronic device includes: if a preset number of images are acquired from the shooting device through the MSMF acquisition backend within a preset time period, then the MSMF acquisition backend is determined as the target acquisition backend; if a preset number of images are not acquired from the shooting device through the MSMF acquisition backend within the preset time period, then the DSHOW acquisition backend is determined as the target acquisition backend.

[0103] Specifically, the electronic device can be configured with an MSMF acquisition backend and a DSHOW acquisition backend. The MSMF acquisition backend is for the new generation multimedia interface of Windows systems, enabling access to the camera device and video stream parsing. The DSHOW acquisition backend is for the older multimedia interface of Windows systems, providing better compatibility with older camera hardware. Both types of video acquisition backends can output captured image frames, which can be used for subsequent preprocessing and portrait segmentation.

[0104] Turn on the USB camera and first enable the MSMF capture backend to read captured image frames. A preset time period is set; check if a preset number of images can be read within this period. If so, the MSMF capture backend can be selected as the target capture backend. For example, the preset number could be 10 frames. If the MSMF capture backend cannot read the preset number of images within the preset time period, switch to the DSHOW capture backend as the target capture backend. If the DSHOW capture backend also fails, a prompt message can be sent to the user, prompting them to check the video capture backend configuration. A video capture backend failure indicates that the current video capture backend has issues such as device incompatibility, hardware disconnection, or driver malfunction.

[0105] In this embodiment, the MSMF backend is prioritized. After opening, it attempts to read 10 frames for verification. If it continues to fail, it automatically falls back to the DSHOW backend. The new generation of multimedia acquisition interface can be used first. When the interface is not compatible, it automatically switches to the video acquisition backend with stronger compatibility, which improves the compatibility of different camera hardware models and ensures the stable acquisition of captured image frames.

[0106] In this embodiment, screen image frames and captured image frames can be acquired separately at various moments during the teacher's lecture, resulting in images containing clear courseware content and images containing people. The electronic device can be equipped with at least one execution provider component. This component determines the hardware required for image segmentation processing, thereby performing image segmentation on the captured image frames to obtain segmented images. The screen image frames and segmented images are then synthesized, and the results from multiple moments are combined to obtain a teaching video. Image segmentation eliminates the need for green screen scenes, reducing scene requirements and increasing the flexibility of video generation. By acquiring screen image frames and captured image frames separately, the courseware content in the teaching video is clear and glare-free, effectively reproducing a realistic classroom scene.

[0107] Figure 7 A flowchart illustrating the method for recording instructional videos, such as... Figure 7 As shown, a teaching video recording method according to one embodiment of this application may include: Step 710: Acquire screen image frames and captured image frames at the same time; wherein, the screen image frame is an image representing the content of the courseware captured on the screen, and the captured image frame is an image containing people captured by the capturing device; Step 720: Based on the captured image frame, determine a portrait segmentation image using at least one execution provider in the electronic device; wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frame; Step 730: Based on the preset homography matrix, transform the portrait segmentation image corresponding to the time point from the preset camera coordinate system to the preset screen coordinate system to obtain the image after coordinate system transformation; Step 740: Perform a fusion process on the screen image frame corresponding to the time and the image after coordinate system transformation to obtain the composite image frame corresponding to the time; wherein, the composite image frame includes the person and the teaching courseware content; Step 750: Generate a teaching video based on the synthetic image frames corresponding to each time point.

[0108] Steps 710 and 720 have been described in detail above and will not be repeated here. Steps 730-750 will be described in detail below.

[0109] In step 730, a homography matrix is ​​pre-set. A homography matrix is ​​a matrix that characterizes the perspective transformation relationship between two planes and is used to achieve mapping and conversion between different image coordinate systems. In this embodiment, the homography matrix is ​​a 3×3 matrix describing the perspective transformation relationship between two planes, used to align the camera viewpoint to the screen viewpoint. That is, through the homography matrix, the segmented image of a person in the camera coordinate system can be mapped and transformed to the screen coordinate system, so that the spatial position of the person in the image matches the actual position of the person relative to the large screen in the physical scene. The image output by the homography matrix is ​​the image after coordinate system transformation. In this embodiment, the transformation process using the homography matrix is ​​not specifically limited.

[0110] In step 740, the screen image frame represents the courseware content, and the portrait segmentation image represents the teacher. After the two are fused, a composite image frame containing both the person and the courseware content is obtained. For example, the portrait segmentation image can be transformed to the screen coordinate system using a homography matrix, and then the screen image frame and the coordinate-transformed image at the same moment can be alpha-blended to obtain the composite image frame.

[0111] For example, after obtaining the image after coordinate system transformation, the transformed image can be fused with the screen image frame. The fusion process can perform alpha blending operations based on the transparency information carried by the transformed image, superimposing the person area onto the courseware background of the screen image frame to generate a composite image frame containing both the person and the courseware content. The fusion process can achieve pixel-level parallel processing using a multi-threaded parallel traversal interface, thereby improving the computational efficiency of image fusion.

[0112] In step 750, the sequence of synthesized image frames is arranged to generate the teaching video. That is, the synthesized image frames are arranged in chronological order to obtain a smooth teaching video. In this embodiment, a preset homography matrix is ​​used to transform the human image segmentation image from the camera coordinate system to the screen coordinate system to obtain the image after coordinate system transformation. Then, the image after coordinate system transformation is fused with the screen image frame at the same time to obtain the synthesized image frame corresponding to that time. After obtaining the synthesized image frames corresponding to each time, the final teaching video is formed by combining the synthesized image frames at all times.

[0113] In this embodiment, the human image segmentation image is converted from the camera coordinate system to the screen coordinate system by a preset homography matrix to obtain the image after coordinate system transformation. Then, the image after coordinate system transformation is fused with the screen image frame to obtain the composite image frame. The teaching video is generated by using the composite image frames at each time. This can make the position of the person in the composite image fit the spatial position relationship of the real classroom, simulate the visual scene of the real classroom, and achieve the recording effect of the person naturally superimposed on the courseware content.

[0114] In this embodiment, screen image frames and captured image frames can be acquired separately at various moments during the teacher's lecture, resulting in images containing clear courseware content and images containing people. The device can be equipped with at least one execution provider component. This component determines the hardware required for image segmentation processing, thereby performing image segmentation on the captured image frames to obtain segmented images. The screen image frames and segmented images are then synthesized, and the combined results from multiple moments are used to obtain a teaching video. Image segmentation eliminates the need for green screen scenes, reducing scene requirements and increasing the flexibility of video generation. By acquiring screen image frames and captured image frames separately, the courseware content in the teaching video is clear and glare-free, effectively reproducing a realistic classroom scene.

[0115] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0116] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0117] Description of apparatus and devices in embodiments of this application Reference Figure 8 , Figure 8 This is a schematic diagram of the structure of the teaching video recording device 800 provided in the embodiments of this application. The teaching video recording device 800 includes: The image acquisition module 801 is used to acquire screen image frames captured from the large screen device and captured image frames captured from the shooting device at the same time; wherein, the screen image frame is an image representing the content of the teaching courseware captured on the screen, and the captured image frame is an image containing people captured by the shooting device. The portrait segmentation module 802 is used to determine a portrait segmentation image based on a captured image frame and at least one execution provider; wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frame; The video generation module 803 is used to synthesize screen image frames and human portrait segmentation images corresponding to several moments to generate teaching videos.

[0118] In one example, the portrait segmentation module 802 includes: A session creation unit is used to create a model inference session based on at least one execution provider in an electronic device; wherein the model inference session represents an instance object of the model after it is running in memory; The portrait segmentation unit is used to run a preset deep learning model based on the model inference session to obtain the portrait segmentation image corresponding to the captured image frame; wherein, the preset deep learning model is used to perform portrait segmentation.

[0119] In one example, the execution provider in the electronic device includes at least one of CUDA execution provider, DirectML execution provider, and CPU execution provider; The session creation unit is specifically used for: The CUDA execution provider is initialized. If the CUDA execution provider is initialized successfully, a model inference session is created through the CUDA execution provider. If the CUDA execution provider initialization fails, the DirectML execution provider will be initialized. If the DirectML execution provider initialization succeeds, a model inference session will be created through the DirectML execution provider. If the DirectML execution provider initialization fails, the CPU execution provider is initialized, and a model inference session is created through the CPU execution provider.

[0120] In one example, the human face segmentation unit is specifically used for: The model inference session runs a preset deep learning model to obtain the initial segmented image corresponding to the captured image frame; wherein, the initial segmented image is a single-channel grayscale image with the same size as the captured image frame; Gaussian blurring is applied to the initial segmented image corresponding to the captured image frame to obtain the portrait segmented image corresponding to the captured image frame.

[0121] In one example, it also includes: A preprocessing module is used to preprocess the captured image frames to obtain preprocessed captured image frames; wherein, the preprocessing includes scaling processing and horizontal mirror flipping processing; based on the preprocessed captured image frames, a human portrait segmentation image is determined based on at least one execution provider in the electronic device.

[0122] In one example, the image acquisition module 801 is specifically used for: The current acquisition window of the large screen device is determined based on the time and the preset time length; where the current acquisition window represents the time window of the screen image frame used for acquisition. If no screen image frame is captured in the current capture window, the previously captured screen image frame is reused as the current screen image frame.

[0123] In one example, the image acquisition module 801 is specifically used for: The DXGI desktop copy interface is used to capture and process the images on the screen of a large-screen device, obtaining the screen image frames at any given moment.

[0124] In one example, the image acquisition module 801 is specifically used for: The target acquisition backend is determined from all video acquisition backends in the electronic device; wherein, the video acquisition backend is used to read images from the shooting device; The target acquisition backend acquires the captured image frames at any given time from the shooting device.

[0125] In one example, the video capture backend includes at least one of the MSMF capture backend and the DSHOW capture backend; Image acquisition module 801 is specifically used for: If a preset number of images are acquired from the shooting device through the MSMF acquisition backend within a preset time period, then the MSMF acquisition backend is determined as the target acquisition backend. If the preset number of images are not acquired from the shooting device through the MSMF acquisition backend within the preset time period, then the DSHOW acquisition backend will be determined as the target acquisition backend.

[0126] In one example, video generation module 803 is specifically used for: Based on the preset homography matrix, the portrait segmentation image corresponding to the time moment is transformed from the preset camera coordinate system to the preset screen coordinate system to obtain the image after coordinate system transformation; The screen image frame corresponding to a given time and the image after coordinate system transformation are fused to obtain the composite image frame corresponding to that time; the composite image frame includes people and courseware content; Based on the synthetic image frames corresponding to each time point, an instructional video is generated.

[0127] In one example, it also includes: The first encoding module is used to compress the teaching video using a preset hardware encoder if the electronic device is equipped with such a module, and then store the compressed teaching video. The second encoding module is used to compress the teaching video using a pre-packaged software encoder if there is no preset hardware encoder in the electronic device, and then store the compressed teaching video.

[0128] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 900 of this embodiment includes: at least one processor 901 ( Figure 9 (Only one is shown in the diagram), memory 902, and computer program 903 stored in memory 902 and executable on at least one processor 901, wherein processor 901 executes computer program 903 to implement the steps in any of the above-described embodiments of the teaching video recording method.

[0131] Electronic device 900 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 901 and a memory 902. Those skilled in the art will understand that... Figure 9This is merely an example of electronic device 900 and does not constitute a limitation on electronic device 900. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0132] The processor 901 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0133] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may be an external storage device of the electronic device 900, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 900. Furthermore, the memory 902 may include both internal and external storage units of the electronic device 900. The memory 902 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 902 can also be used to temporarily store data that has been output or will be output.

[0134] This application also provides a large-screen device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, it implements the steps in any of the above method embodiments.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0136] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recording instructional videos, characterized in that, include: The system acquires screen image frames captured from a large-screen device and captured image frames captured from a camera device at the same time. The screen image frames are images representing the content of teaching courseware captured on the screen, and the captured image frames are images containing people captured by the camera device. Based on the captured image frame, a portrait segmentation image is determined based on at least one execution provider component, the execution provider component being used to specify hardware for performing image segmentation processing, the portrait segmentation image representing an image of a person segmented from the captured image frame; The screen image frames and portrait segmentation images corresponding to several of the aforementioned times are synthesized to generate an instructional video.

2. The method according to claim 1, characterized in that, The step of determining a portrait segmentation image based on the captured image frame and using at least one execution provider includes: A model inference session is created based on at least one execution provider; wherein the model inference session represents an instance object of the model after it is running in memory; Based on the model inference session, a preset deep learning model is run to obtain the portrait segmentation image corresponding to the captured image frame; wherein, the preset deep learning model is used for portrait segmentation.

3. The method according to claim 2, characterized in that, The execution provider includes at least one of CUDA execution provider, DirectML execution provider, and CPU execution provider; The step of creating a model inference session based on at least one execution provider includes: The CUDA execution provider is initialized. If the CUDA execution provider is successfully initialized, the model inference session is created through the CUDA execution provider. If the CUDA execution provider initialization fails, the DirectML execution provider is initialized; if the DirectML execution provider initialization succeeds, the model inference session is created through the DirectML execution provider. If the DirectML execution provider fails to initialize, the CPU execution provider is initialized, and the model inference session is created through the CPU execution provider.

4. The method according to claim 2, characterized in that, The process of running a preset deep learning model based on the model inference session to obtain a portrait segmentation image corresponding to the captured image frame includes: Based on the model inference session, a preset deep learning model is run to obtain the initial segmented image corresponding to the captured image frame; wherein, the initial segmented image is a single-channel grayscale image with the same size as the captured image frame; Gaussian blur processing is applied to the initial segmented image corresponding to the captured image frame to obtain the portrait segmented image corresponding to the captured image frame.

5. The method according to claim 1, characterized in that, The method further includes: The captured image frames are preprocessed to obtain preprocessed captured image frames; wherein, the preprocessing includes scaling and horizontal mirror flipping. Based on the preprocessed captured image frames, a portrait segmentation image is determined using at least one execution provider.

6. The method according to claim 1, characterized in that, The acquisition of screen image frames captured from the large-screen device at the same time includes: Based on the stated time and a preset time length, the current acquisition window of the large screen device is determined; wherein, the current acquisition window represents the time window used to acquire screen image frames at the stated time. If no screen image frame is captured in the current capture window, the previously captured screen image frame is reused as the screen image frame at the current moment.

7. The method according to claim 1, characterized in that, The acquisition of screen image frames captured from the large-screen device at the same time includes: The screen of the large-screen device is captured and processed using the DXGI desktop copy interface to obtain the screen image frame at the specified moment.

8. The method according to claim 1, characterized in that, Acquire image frames captured from the imaging device at the same time, including: The target acquisition backend is determined from all video acquisition backends; wherein, the video acquisition backend is used to read images from the shooting device; The target acquisition backend acquires the captured image frame at the specified moment from the capturing device.

9. The method according to claim 8, characterized in that, The video acquisition backend includes at least one of MSMF acquisition backend and DSHOW acquisition backend; The step of determining the target acquisition backend from all video acquisition backends includes: If a preset number of images are acquired from the shooting device through the MSMF acquisition backend within a preset time period, then the MSMF acquisition backend is determined as the target acquisition backend; If a preset number of images are not acquired from the shooting device through the MSMF acquisition backend within a preset time period, then the DSHOW acquisition backend is determined as the target acquisition backend.

10. The method according to claim 1, characterized in that, The step of synthesizing screen image frames and portrait segmentation images corresponding to several said times to generate a teaching video includes: Based on the preset homography matrix, the portrait segmentation image corresponding to the specified time is converted from the preset camera coordinate system to the preset screen coordinate system to obtain the image after coordinate system transformation; The screen image frame and the image after coordinate system transformation corresponding to the specified time are fused to obtain the composite image frame corresponding to the specified time; wherein, the composite image frame includes people and teaching courseware content; The teaching video is generated based on the synthesized image frames corresponding to each of the stated times.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: If the large-screen device is equipped with a preset hardware encoder, the teaching video is compressed using the preset hardware encoder, and the compressed teaching video is stored. If the large-screen device does not have a preset hardware encoder, the teaching video is compressed using a pre-packaged software encoder, and the compressed teaching video is stored.

12. A teaching video recording device, characterized in that, include: The image acquisition module is used to acquire screen image frames captured from a large-screen device and captured image frames captured from a shooting device at the same time. The screen image frames are images representing the content of teaching courseware captured on the screen, and the captured image frames are images containing people captured by the shooting device. A portrait segmentation module is used to determine a portrait segmentation image based on the captured image frame and at least one execution provider, wherein the execution provider component is used to specify the hardware for performing image segmentation processing, and the portrait segmentation image represents an image of a person segmented from the captured image frame; The video generation module is used to synthesize screen image frames and human portrait segmentation images corresponding to several times to generate teaching videos.

13. A large-screen device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 11.