Video synthesis method and device, intelligent interactive panel and storage medium
By dynamically adjusting the overlapping positions of images in the video composite, the problem of upper-layer video obscuring lower-layer screen operations is solved, achieving clear display of important information and improving the audience experience.
Patent Information
- Application Number
- CN202411781848.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2026-06-05
Smart Images
Figure CN122160471A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video compositing technology, and in particular to a video compositing method, a picture-in-picture position determination method, an apparatus, an intelligent interactive flat panel, and a storage medium. Background Technology
[0002] With the development of technology, multimedia technology is gradually being integrated into teaching activities. As a form of multimedia, instructional videos can enrich teachers' teaching methods and enhance students' learning interest. Instructional videos can include screen recordings, camera videos, and composite videos created by overlaying multiple video streams, among others.
[0003] For example, a teacher's video can be embedded into a specific part of a screen recording of the teacher demonstrating on screen, creating a composite video for students to watch online. This "picture-in-picture" approach simulates a face-to-face teaching environment, enhancing the student learning experience.
[0004] However, the position of the upper-layer camera video feed in the composite video is fixed. When screen operations displayed in the lower-layer screen recording video fall into that position, the screen operations will be obscured. In short, the fixed position of the upper video feed will continuously obscure the lower video feed, affecting the display of important information. Summary of the Invention
[0005] This application provides a video compositing method, a picture-in-picture position determination method, an apparatus, a smart interactive flat panel, and a storage medium, which can solve the technical problem of "the upper video screen being fixed in position, continuously obscuring the lower video screen". The technical solution provided by this application is as follows:
[0006] In a first aspect, embodiments of this application provide a video synthesis method, applied to a first display device, comprising:
[0007] The screen operation position of the first screen operation of the first display device in the current display screen is obtained, and it is determined whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0008] If the screen operation position falls within the first designated area, the display position of the first display window is changed from the first designated area to the second designated area so that the screen operation position does not fall within the second designated area.
[0009] The target sub-screen is loaded into the first display window, and the next screen of the current display screen is loaded into the second display window to obtain the composite screen; wherein, the first display window is located on top of the second display window, and the second display window is used to display the next screen of the current display screen; the composite screen is transmitted from the first display device to the second display device for display.
[0010] Secondly, embodiments of this application provide a method for determining the position of a composite video picture-in-picture, applied to a display device, including:
[0011] The screen operation position of the first screen operation of the display device in the current display screen is obtained, and it is determined whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0012] If the screen operation position falls within the first designated area, the display position of the first display window is changed from the first designated area to the second designated area so that the screen operation position does not fall within the second designated area.
[0013] Thirdly, embodiments of this application provide a video synthesis apparatus, including:
[0014] The screen operation information acquisition module is used to acquire the screen operation position of the first screen operation of the first display device in the current display screen, and to determine whether the screen operation position falls into the first designated area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0015] The region setting module is used to change the display position of the first display window from the first specified region to the second specified region when the screen operation position falls into the first specified region, so that the screen operation position does not fall into the second specified region.
[0016] The image compositing module is used to load the target sub-image into the first display window and load the next image of the current display image into the second display window to obtain the composite image; wherein, the first display window is located on top of the second display window, and the second display window is used to display the next image of the current display image; the composite image is transmitted from the first display device to the second display device for display.
[0017] Fourthly, embodiments of this application provide a device for determining the position of a picture-in-picture in a composite video, comprising:
[0018] The screen operation information acquisition module is used to acquire the screen operation position of the first screen operation of the display device in the current display screen, and to determine whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0019] The region setting module is used to change the display position of the first display window from the first specified region to the second specified region when the screen operation position falls into the first specified region, so that the screen operation position does not fall into the second specified region.
[0020] Fifthly, embodiments of this application provide an intelligent interactive flat panel, comprising: a processor and a memory; the memory stores a computer program, the computer program being adapted to be loaded by the processor and executed as the video synthesis method described in the first aspect or the method for determining the picture-in-picture position of the synthesized video described in the second aspect.
[0021] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor and executing the video compositing method as described in the first aspect or the method for determining the picture-in-picture position of the composite video as described in the second aspect.
[0022] In this embodiment, during the compositing of a video (referred to as the upper-layer video) and a screen recording of a user's screen, the upper-layer video can be superimposed on a first designated area of the screen recording to obtain the composite video. That is, the upper-layer video is displayed on top of the screen recording in a "picture-in-picture" mode. If a user performs a screen operation within the first designated area, and the upper-layer video is still superimposed on the first designated area of the screen recording, the upper-layer video in the composite video will obscure the user's screen operation contained in the lower screen recording. Therefore, when the screen operation is determined to be within the first designated area, the position of the overlapping area in the subsequent composite video can be adjusted, i.e., the first designated area can be switched to a second designated area. Then, based on the second designated area, the upper-layer video is superimposed on top of the screen recording to obtain the subsequent composite video. This avoids obscuring the user's screen operation in the composite video and clearly displays important information from the screen recording.
[0023] Furthermore, in this embodiment, the duration of user screen operations occurring within a first designated area can be obtained. If the duration reaches a first preset duration, the first designated area is switched to a second designated area. This prevents accidental user operations from causing the position of the upper-layer video image superimposed on the screen recording video image to be switched, thereby avoiding distracting the viewer's attention and causing visual confusion, and ensuring a good viewing experience.
[0024] Furthermore, since users may perform multiple screen operations within the first designated area, the determination of the continuity of touch operations can be delayed until a second preset duration. In other words, if the interval between any two screen operations does not exceed the second preset duration, it can be considered a continuous touch operation, and the total duration of this touch operation is recorded as the continuous operation duration of the current user's screen operation. This avoids frequent calculations and determinations of the continuous operation duration in situations involving rapid, multiple touch operations such as writing by hand or hand tremors, thus saving computing resources and ensuring timely switching of overlapping screen areas.
[0025] In this embodiment, during the process of compositing the upper-layer video and the screen-recorded video based on the second designated area, it is possible to detect whether a new user screen operation occurs in the first designated area within a third preset time period. If no new user screen operation occurs, it indicates that compositing the upper-layer video and the screen-recorded video based on the first designated area will not obscure the user screen operation in the screen-recorded video, and the overlapping area can be restored to the first designated area. Since viewers habitually look at the default rendering area of the upper-layer video, overlaying the upper-layer video on the first designated area of the screen-recorded video is more easily noticed by viewers, thereby improving the display effect of the composite video and enhancing the viewer experience.
[0026] In this embodiment, user screen operation can be multi-point operation. If any operation point is located in a first designated area, the user's current screen operation is determined to be located in the first designated area. For example, if the user screen operation is a multi-touch operation, the user's touch point may vary due to differences in hand position, size, or posture. If any touch point falls into the first designated area, the screen operation position can be determined to be located in the first designated area, thereby triggering the switching of overlapping screen areas and improving the flexibility of switching operations.
[0027] In this embodiment, the images can be rendered using OpenGL (Open Graphics Library). For example, a texture can be created for each video, and the images from each video can be mapped onto the corresponding texture for rendering to obtain a composite image. In this way, OpenGL can efficiently process a large number of video images by rendering images from multiple videos as texture data, improving the rendering efficiency of multi-channel video and making it suitable for real-time video compositing tasks.
[0028] In this embodiment, visual feedback can also be provided to the user, such as marking overlapping areas on the screen with a semi-transparent highlight effect or border, to remind the user whether their screen operations will be obscured in the composite video. If the user needs to perform screen operations at the marked location, the user can be certain that their screen operations will be obscured in the subsequent composite video. The user can actively issue switching commands through gestures such as double-tap, long press, or swipe to quickly switch the overlapping areas of the composite video. In this way, based on the user's actively issued switching commands, the position of the upper-layer video in the composite video can be switched at any time, preventing the user's screen operations in the lower-layer screen recording video from being obscured in advance, and clearly displaying important information from the screen recording video. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1A A schematic diagram of a screen recording video provided in an embodiment of this application;
[0031] Figure 1B A schematic diagram of another screen recording video provided in an embodiment of this application;
[0032] Figure 2 A schematic diagram of a video image provided in an embodiment of this application;
[0033] Figure 3A A schematic diagram of a synthesized video provided in an embodiment of this application;
[0034] Figure 3B A schematic diagram of a synthesized video provided in an embodiment of this application;
[0035] Figure 4 A flowchart illustrating a video synthesis method provided in an embodiment of this application;
[0036] Figure 5 A flowchart illustrating another video synthesis method provided in an embodiment of this application;
[0037] Figure 6A A schematic diagram of a synthesized video provided in an embodiment of this application;
[0038] Figure 6B A schematic diagram of a synthesized video provided in an embodiment of this application;
[0039] Figure 7 This is a schematic diagram of the structure of a video synthesis device provided in an embodiment of this application;
[0040] Figure 8 A schematic diagram of the structure of a device for determining the position of a composite video picture-in-picture provided in an embodiment of this application;
[0041] Figure 9 This is a schematic diagram of the structure of a smart interactive flat panel provided in an embodiment of this application. Detailed Implementation
[0042] With the improvement of computer processing power and the advancement of image processing algorithms, multi-channel video synthesis technology has been widely used in various fields.
[0043] For example, in online education, teachers can create more vivid and intuitive teaching videos by combining multiple video feeds from different sources into a single frame. Specifically, by combining the teacher's camera footage with screen recordings, the teacher can demonstrate on the screen of electronic devices such as smart interactive whiteboards and smart blackboards. (See also...) Figure 1A The teacher selects file 5 on the screen by long-pressing. By recording the screen of the electronic device, a screen recording of the teacher's demonstration can be obtained. While the teacher is operating the screen interface, you can refer to... Figure 2 The teacher was filmed using cameras or other video recording equipment, resulting in video footage of the teacher. This can then be viewed... Figure 3A By embedding the teacher's video recording into a designated location within the teacher's screen recording, such as the rectangular area in the upper right corner, a composite video can be created for students to view online. This "picture-in-picture" approach simulates a face-to-face teaching environment, enhancing the student learning experience.
[0044] For example, in the field of game live streaming, game streamers can create more vivid and intuitive game live stream videos by combining multiple video feeds from different sources into a single frame. Specifically, taking the combination of the streamer's camera footage and screen recordings as an example, the streamer can perform game operations on the screen of a personal computer, mobile phone, or other electronic device. Furthermore, by recording the screen of this electronic device, a screen recording of the streamer's gameplay can be obtained. Simultaneously, a camera can be used to film the streamer, resulting in camera footage. This camera footage is then embedded into a designated location within the streamer's screen recording, such as the rectangular area in the upper right corner, to create a composite video for viewers to watch online. This "picture-in-picture" approach simulates a face-to-face interaction between the streamer and the audience. Viewers not only see the streamer's game interface but also their real facial expressions and body language, enhancing viewer engagement and interactivity.
[0045] However, as shown in Figure 3, the position of the upper-layer video (such as the camera video mentioned above) in the composite video is fixed; that is, the overlapping position is fixed, and it will continuously obscure the content of the lower-layer video (such as the screen recording video) at that position. When the lower-layer video contains important information such as screen operations at the overlapping position, this important information cannot be displayed in the composite video because it is obscured by the upper-layer video.
[0046] Based on this, this application proposes a video synthesis method. This method can dynamically adjust the overlapping position of multiple video streams during synthesis according to the screen operation position, so as to avoid the screen operation position in the lower layer screen recording video being obscured by the upper layer video in the synthesized video, thereby clearly displaying the important information in the screen recording video in the synthesized video.
[0047] It is worth noting that the upper-layer video can be any video frame, including videos shot in real-time by camera equipment, pre-produced videos, or preset still images; this application does not impose any restrictions on this. It can be understood that video is composed of a series of consecutive video frames (i.e., a single still image frame). For example, a frame in the upper-layer video can be called an upper-layer video frame, and correspondingly, a frame in the lower-layer video can be called a lower-layer video frame. Therefore, the process of combining multiple video streams to obtain a composite video can be understood as a continuous application of image compositing processing.
[0048] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0049] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0050] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0051] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0052] Figure 4 This is a schematic flowchart illustrating a video synthesis method provided in an embodiment of this application. The following will describe the method in conjunction with specific implementation details. Figure 4 The processing flow shown is explained in detail below:
[0053] S102, obtain the screen operation position of the first screen operation in the current display screen of the first display device, and determine whether the screen operation position falls into the first specified area corresponding to the first display window.
[0054] The first display window is located above the second display window. The first display window is used to display the target sub-screen in the composite screen, and the second display window is used to display the next screen after the current screen.
[0055] It can be understood that screen recording video from a display device consists of a series of consecutive screenshots, with each screenshot corresponding to a display frame. When a screen operation occurs on the display device's screen, the display frame corresponding to a certain moment in that operation can be called the current display frame, and the display frames corresponding to subsequent moments can be called the next frame after the current display frame.
[0056] In practice, the user-operated electronic device includes a display screen (i.e., a display device), and the operating system of the electronic device can monitor the screen operations received by the electronic device. The aforementioned screen operations can be any type of screen operation, such as touching the screen, operating the screen with a mouse, or operating the screen with buttons; this application does not impose any restrictions on this.
[0057] For example, the user's electronic device is a smart interactive tablet running Android, and the user's screen operation involves touching the touchscreen with their finger or stylus. The smart interactive tablet can listen to user touch operations through Android's native touch API (Application Programming Interface). Specifically, a listener is registered in the Phone Window Manager at the Framework layer to capture touch events (Motion Events). When the user touches the smart interactive tablet's screen, the MotionEvent records touch information such as the position and number of touch points. The position of the touch point can be represented by its coordinates (x, y) on the screen. Then, the function `Rect.contains(x, y)` can be used to check if the touch point coordinates (x, y) are within the boundaries of a first specified area.
[0058] It is worth mentioning that the electronic device operated by the user can use any operating system such as Android, iOS, Windows, or Linux, and this application does not impose any restrictions on this.
[0059] For ease of description, the display device operated by the user can be referred to as the first display device, and any screen operation performed by the user on the screen of the first display device can be referred to as the first screen operation.
[0060] In one embodiment, the first screen operation can be a multi-point operation. The specific processing of step S102 may include: obtaining the positions of multiple operation points of the first screen operation, and determining whether the position of each operation point falls within a first specified area; if any operation point falls within the first specified area, then it is determined that the screen operation position falls within the first specified area.
[0061] In implementation, the screen operated by the user can be a touchscreen that supports multi-touch operations. In a multi-touch environment, the touch point of a user's single touch operation may vary depending on the hand position, size, or posture. If the first screen operation is a multi-touch operation, touch information such as the position of multiple touch points can be recorded through Motion Events. As long as the position of any one operation point is within a first specified area, the position of the entire screen operation can be determined to be within the first specified area. This triggers the switching of the display position of the first display window (i.e., the overlapping area of the upper and lower video frames), improving the flexibility of the overlapping area switching operation.
[0062] S104, if the screen operation position falls within the first designated area, the display position of the first display window is changed from the first designated area to the second designated area.
[0063] In implementation, during the compositing of the upper-layer video and the screen recording of the user's screen, the upper-layer video can be overlaid on top of a first designated area of the screen recording to obtain the composite video. That is, the upper-layer video is displayed on top of the screen recording in a "picture-in-picture" mode. If the user performs a screen operation within the first designated area, and the upper-layer video is still overlaid on top of that area, the upper-layer video in the composite video will obscure the user's screen operation within the lower screen recording. Therefore, when the screen operation is determined to be within the first designated area, the position of the overlapping area in the subsequent composite video can be adjusted, switching the first designated area to a second designated area so that the current screen operation does not fall within the second designated area. This way, when compositing the upper-layer video and the screen recording based on the second designated area, the user's screen operation in the composite video is not obscured, thus clearly displaying important information from the screen recording.
[0064] S106, load the target sub-screen into the first display window, and load the next screen of the current display screen into the second display window to obtain the composite screen.
[0065] In implementation, by switching the first designated area to the second designated area, and then overlaying the upper-layer video image onto the screen-recorded video image based on the second designated area, the resulting composite video image is obtained. This avoids obscuring user screen operations located in the first designated area from the composite video, ensuring that important information from the screen-recorded video is clearly displayed.
[0066] For example, the first designated area could be the upper right corner of the screen recording video (see...). Figure 3A The second designated area is the top left corner of the screen recording video. After switching between overlapping areas, in Figure 6A In the composite image shown, the image from the upper-layer video is rendered onto a certain area in the upper left corner of the screen recording video.
[0067] For example, the first designated area could be the upper right corner of the screen where the video is recorded (see...). Figure 3B The second designated area is the lower right corner of the screen recording video. After switching between overlapping areas of the image, in Figure 6B In the composite image shown, the image from the upper-layer video is rendered onto a certain area in the lower right corner of the screen recording video.
[0068] It should be noted that the first designated area and the second designated area described in the various embodiments of this application can be any area in the screen-recorded video, or the first designated area and the second designated area can be any area in the corresponding composite image, and this application does not impose any restrictions on this.
[0069] The user-operated display device, i.e., the first display device, can transmit the composite video to other display devices for playback after obtaining it. For ease of description, any display device used to play the composite video, i.e., any display device used to display the composite video, can be referred to as the second display device.
[0070] For example, the first display device could be the teacher's smart interactive whiteboard, and the second display device could be the student's smart interactive whiteboard. When the teacher demonstrates on their smart interactive whiteboard, the whiteboard can continuously record the screen and generate composite images, then send a series of composite images to the student's smart interactive whiteboard in real time so that the student's smart interactive whiteboard can display these composite images.
[0071] It is worth mentioning that the first display device can transmit the composite image to the second display device during the process of generating the composite video, or it can transmit the composite image to the second display device after the video is composited. This application does not impose any restrictions on this.
[0072] In one embodiment, the specific processing of step S106 may include: setting the second specified area as the texture coordinates of the first display window; calling the Open Graphics Library to render the images of the upper-layer video and the lower-layer video according to the corresponding texture coordinates to obtain a composite image.
[0073] In implementation, the images can be rendered using OpenGL (Open Graphics Library). For example, a separate texture can be created for each video, and the images from each video can be mapped onto the corresponding texture for rendering to obtain the composite image. In this way, OpenGL can efficiently process a large number of video images by rendering images from multiple videos as texture data, improving the rendering efficiency of multi-channel video and making it suitable for real-time video compositing tasks.
[0074] For example, after setting the display position of the first display window (i.e. the overlapping area of the composite image) to the second specified area, the matrix coordinates of the texture of the upper-layer video in OpenGL can be modified, and the matrix can be applied to the vertex coordinates of the texture of the upper-layer video to realize the rendering operation of the composite image.
[0075] It is worth mentioning that the second specified region can be a preset region independent of the first specified region, or it can be obtained by moving the texture corresponding to the first specified region using the Matrix.translateM method and / or scaling the texture corresponding to the first specified region using the Matrix.scaleM method. This application does not impose any restrictions on this.
[0076] It should be noted that "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.
[0077] In one embodiment, see Figure 5 If the screen operation position is within the first designated area, before changing the display position of the first display window from the first designated area to the second designated area, the video compositing method provided in this application may include:
[0078] S202, Obtain the screen operation position of the user's first screen operation.
[0079] S204, determine whether the screen operation position is located in the first specified area.
[0080] In practice, the specific processing of steps S202 and S204 can be found in step S102, and will not be repeated here.
[0081] S206, Obtain the duration of continuous operation of the first screen operation within the first specified area.
[0082] In one embodiment, a touch event can typically include the following operation types: MotionEvent.ACTION_DOWN, which indicates that the screen is pressed and can be triggered once in a single touch operation; MotionEvent.ACTION_MOVE, which indicates that the finger moves on the screen and can be triggered zero or more times in a single touch operation; and MotionEvent.ACTION_UP, which indicates that the finger leaves the screen and can be triggered once in a single touch operation.
[0083] In one embodiment, step S206 may specifically include: obtaining the entry time when the first screen operation enters the first designated area and the exit time when it leaves the first designated area; calculating the time difference between the exit time and the entry time to obtain the duration of the first screen operation within the first designated area.
[0084] For example, see Figure 1AWhen a user long-presses a location within the first specified area, one `MotionEvent.ACTION_DOWN` event and one `MotionEvent.ACTION_UP` event are triggered. The corresponding time can be obtained using the `System.currentTimeMillis()` method when each event occurs. Therefore, the time of the `MotionEvent.ACTION_DOWN` event is the entry time when the long-press operation enters the first specified area, and the time of the `MotionEvent.ACTION_UP` event is the exit time when the long-press operation leaves the first specified area.
[0085] For example, see also Figure 1B When the user swipes on the screen, one MotionEvent.ACTION_DOWN event, multiple MotionEvent.ACTION_MOVE events, and one MotionEvent.ACTION_UP event are triggered. Specifically, when the MotionEvent.ACTION_DOWN event occurs, the touch point has not yet entered the first designated area; it enters the first designated area during the swipe touch. The intersection of the touch event and the first designated area is point A. Figure 1B (Not shown), then the time when the touch operation reaches point A is the entry time when the touch operation enters the first designated area. It can be understood that if the user's finger leaves the screen within the first designated area after the swipe operation enters it, the time when the MotionEvent.ACTION_UP event occurs is the exit time when the touch operation leaves the first designated area; if the user swipes into the first designated area and then swipes out of it, the intersection point of the touch event with the first designated area during the swipe-out process is point B (…). Figure 1B (not shown), then the time when the touch operation reaches point B is the departure time when the touch operation leaves the first designated area.
[0086] The above examples demonstrate how to accurately obtain the entry time when the first screen operation enters the first designated area and the exit time when it leaves the first designated area. Furthermore, by calculating the time difference between the exit time and the entry time, the duration of the first screen operation within the first designated area can be determined.
[0087] In one embodiment, a user may perform multiple screen operations within a first designated area. The continuity determination of the touch operations can be delayed to a second preset duration. That is, if the time interval between any two screen operations does not exceed the second preset duration (e.g., 100 milliseconds), then the two screen operations are considered as a single continuous screen operation. Accordingly, the processing in step S206 may specifically include: obtaining the entry time when the first screen operation enters the first designated area and the exit time when it leaves the first designated area; determining whether a second screen operation enters the first designated area within the second preset duration after the exit time; if so, identifying the second screen operation as part of the first screen operation and updating the time when the second screen operation leaves the first designated area to the exit time corresponding to the first screen operation; calculating the time difference between the exit time and the entry time of the first screen operation to obtain the continuous operation duration.
[0088] In implementation, for ease of description, a screen operation occurring after the first screen operation can be referred to as the second screen operation. If the time interval between the entry time corresponding to the second screen operation and the exit time corresponding to the first screen operation does not exceed a second preset duration, then the second screen operation and the first screen operation can be considered as a continuous touch operation, that is, the second screen operation is determined as part of the first screen operation. In other words, the total duration of this continuous touch operation can be recorded as the continuous operation duration of the first screen operation. In this way, in cases where multiple touch operations occur rapidly, such as writing by stroke or hand tremors, the calculation and judgment of the continuous operation duration can be avoided frequently, thereby saving computing resources and ensuring timely switching of overlapping screen areas.
[0089] In one embodiment, the continuity determination of a touch event can be delayed to a second preset duration using the debounce function. That is, when the first screen operation leaves the first designated area, the time difference between the departure time and the entry time of the first screen operation is not calculated. If, within the second preset duration after the first screen operation leaves the first designated area, another touch operation enters the first designated area, it can be considered that the first screen operation has not left the first designated area, and the determination of the duration of the first screen operation is removed.
[0090] In one embodiment, the duration can be determined using the `Handler.postDelayed()` method and removed using the `Handler.removeCallbacks()` method. Upon each touch event detected, it can be first determined whether the current touch operation is already within the first specified area, and then different `Runnable` objects can be submitted to the `Handler` accordingly. These `Runnable` objects include options for "moving out of the overlapping screen area" or "moving into the overlapping screen area". After the `Runnable` objects in the `Handler` have completed their execution, the corresponding overlapping screen area switching operation is triggered.
[0091] S208, determine whether the continuous operation duration has reached the first preset duration.
[0092] In one embodiment, if the continuous operation time reaches a first preset time, steps S210 and S212 are executed; otherwise, steps S214 and S216 are executed.
[0093] S210, change the display position of the first display window from the first designated area to the second designated area.
[0094] S212, keep the display position of the first display window in the first specified area.
[0095] S214, load the target sub-screen into the first display window, and load the next screen of the current display screen into the second display window to obtain the composite screen.
[0096] In practice, the specific processing of step S210 can be found in step S104, and the specific processing of step S214 can be found in step S106. This application will not elaborate further here.
[0097] In implementation, the user's display device can continuously monitor screen operations occurring on the display screen and calculate the time interval between the current time and the entry time when the first screen operation enters the first designated area. Once this time interval reaches a first preset duration, it is considered that the user is performing a valid screen operation in the first designated area, and the display position of the first display window can be modified. Otherwise, it is considered that the user has performed an erroneous operation in the first designated area, and the display position of the first display window will not be modified.
[0098] In this way, by setting a first preset duration, such as 3 seconds, it is possible to identify user misoperation, thereby preventing the position of the upper video image superimposed on the screen recording video image from being switched due to user misoperation, avoiding the composite video image from distracting the audience's attention and causing visual confusion, and ensuring the audience's viewing experience.
[0099] In one embodiment, after step S104, the video synthesis method may further include: obtaining the departure time when the first screen operation leaves the first designated area; determining whether a third screen operation enters the first designated area within a third preset time after the departure time; if not, restoring the display position of the first display window to the first designated area.
[0100] In implementation, by presetting a third preset duration, such as 5 seconds, it can promptly identify whether the user has not performed any screen operation within the default first designated area for an extended period. If, after changing the display position of the first display window from the first designated area to the second designated area, no user screen operation occurs within the first designated area for a prolonged period, it indicates that if the upper-layer video and the screen recording video are composited based on the first designated area, the user's screen operation in the screen recording video will not be obscured. The overlapping area can then be restored to the first designated area. Subsequently, the subsequent frames of the upper-layer video are overlaid on top of the subsequent frames of the screen recording video, according to the first designated area, to obtain the composite image. Since the upper-layer video is displayed using the first designated area for an extended period, viewers may habitually look towards that area. Therefore, promptly restoring the position of the upper-layer video overlaid on the screen recording video to the first designated area makes it easier for viewers to notice, thereby improving the display effect of the composite video and enhancing the viewer experience.
[0101] It should be noted that the first preset duration, the second preset duration, and the third preset duration mentioned above can be set and adjusted according to the actual application scenario, and this application does not limit their values.
[0102] In one embodiment, the video synthesis method may further include: displaying a marker of the first display window; the marker being used to indicate the display position of the first display window to a user; receiving a position switching instruction for the first display window; and switching the display position of the first display window according to the position switching instruction.
[0103] In other words, the user can display a marker of the current overlapping area on the user's screen; the marker is used to prompt the user whether the current overlapping area needs to be switched; the user receives a switching instruction for the overlapping area and switches the current overlapping area according to the switching instruction.
[0104] In implementation, a marker can be displayed on the user's screen indicating the current overlapping area of the video feed, providing visual feedback. For example, by marking the overlapping area with a semi-transparent highlight or border, the user can be alerted to whether their current touch operation falls within the overlapping area of the composite video. This encourages users to avoid entering the marked area and minimizes the need to frequently switch between upper-layer video frames, thus preventing disruption to viewer attention and visual confusion.
[0105] Furthermore, by providing visual feedback to users, it's possible to alert them whether their screen actions will be obscured in the composite video. If a user needs to perform screen actions at a marked location, they can be certain their actions will be obscured in the subsequent composite video. Users can proactively issue switching commands via double-tap, long-press, or swipe gestures to quickly change the overlapping areas of the composite video. In this way, based on user-initiated switching commands, the position of the upper-layer video within the composite video can be changed at any time, proactively preventing obscuring of user screen actions in the lower-layer screen recording video and clearly displaying important information from the screen recording video.
[0106] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.
[0107] For example, in one embodiment, a technical feature is described: determining the validity of a user's screen operation by using a first preset duration includes: determining whether the duration of the first screen operation within a first designated area reaches the first preset duration; if the duration of the operation does not reach the first preset duration, it indicates that the first screen operation is likely a mistake, and the overlapping area of the composite image (i.e., the display position of the first display window) can be maintained as the first designated area. In another embodiment, another technical feature is described: if no third screen operation enters the first designated area within a third preset duration, the display position of the first display window is restored to the first designated area.
[0108] Since the above two technical features are not contradictory, those skilled in the art, after reading this application specification, should be able to conceive that an implementation method that simultaneously possesses both features is also an optional implementation method. That is, after modifying the display position of the first display window from the first designated area to the second designated area, if a third screen operation enters the first designated area within a third preset time period, it is further determined whether the continuous operation time of the third screen operation within the first designated area reaches the first preset time period. If the continuous operation time does not reach the first preset time period, it indicates that the third screen operation is likely an invalid erroneous operation, and it can be regarded as no third screen operation entering the first designated area, and the display position of the first display window is restored to the first designated area. If the continuous operation time reaches the first preset time period, it indicates that the third screen operation within the first designated area is a valid screen operation, and the display position of the first display window can be maintained at the second designated area.
[0109] Based on the same technical concept, embodiments of this application also provide a method for determining the picture-in-picture position of a synthesized video, including:
[0110] The screen operation position of the first screen operation of the display device in the current display screen is obtained, and it is determined whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0111] If the screen operation position falls within the first designated area, the display position of the first display window is changed from the first designated area to the second designated area so that the screen operation position does not fall within the second designated area.
[0112] In one embodiment, before changing the display position of the first display window from the first designated area to the second designated area when the screen operation position falls within the first designated area, the above determination method further includes:
[0113] The duration of the first screen operation within the first specified area is obtained, and it is determined whether the duration of the continuous operation reaches a first preset duration.
[0114] If the continuous operation time reaches the first preset time, the display position of the first display window will be changed from the first designated area to the second designated area.
[0115] In one embodiment, obtaining the duration of the first screen operation within a first specified area specifically includes:
[0116] Obtain the entry time when the first screen operation enters the first designated area, and the exit time when it leaves the first designated area;
[0117] Determine whether a second screen operation enters the first designated area within a second preset time period after the departure time;
[0118] If the second screen operation enters the first designated area, the second screen operation is determined to be part of the first screen operation, and the time when the second screen operation leaves the first designated area is updated to the departure time.
[0119] The duration of continuous operation is obtained by calculating the time difference between the departure time and the entry time.
[0120] In one embodiment, after changing the display position of the first display window from the first designated area to the second designated area, the above determination method further includes:
[0121] Obtain the departure time when the first screen operation leaves the first designated area;
[0122] Determine whether a third screen operation enters the first designated area within a third preset time period after the departure time;
[0123] If no third screen operation enters the first designated area, the display position of the first display window is restored to the first designated area.
[0124] In one embodiment, the first screen operation is a multi-point operation; obtaining the screen operation position of the first screen operation on the display device in the current display screen, and determining whether the screen operation position falls within the first designated area corresponding to the first display window, includes:
[0125] Obtain the positions of multiple operation points of the first screen operation, and determine whether the position of each operation point falls within the first designated area;
[0126] If any of the operation points falls within the first designated area, then the screen operation position is determined to fall within the first designated area.
[0127] In one embodiment, the above determination method further includes:
[0128] Receive a position switching instruction for the first display window, and switch the display position of the first display window according to the position switching instruction.
[0129] In implementation, a marker for the first display window can be displayed on the screen of the display device; this marker is used to indicate the display position of the first display window to the user. The user can then actively operate the display device to switch the display position of the first display window based on the marker displayed on the screen.
[0130] Based on the same technical concept, this application also provides a video synthesis apparatus, which can be found in [reference needed]. Figure 7 ,include:
[0131] The screen operation information acquisition module is used to acquire the screen operation position of the first screen operation of the first display device in the current display screen, and to determine whether the screen operation position falls into the first designated area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0132] The region setting module is used to change the display position of the first display window from the first designated region to the second designated region when the screen operation position falls into the first designated region, so that the screen operation position does not fall into the second designated region.
[0133] The image compositing module is used to load the target sub-image into the first display window and load the next image of the current display image into the second display window to obtain a composite image; wherein, the first display window is located on the upper layer of the second display window, and the second display window is used to display the next image of the current display image; the composite image is transmitted from the first display device to the second display device for display.
[0134] In one embodiment, the locale setting module is specifically used for:
[0135] The duration of the first screen operation within the first specified area is obtained, and it is determined whether the duration of the continuous operation reaches a first preset duration.
[0136] If the continuous operation time reaches the first preset time, the display position of the first display window will be changed from the first designated area to the second designated area.
[0137] Furthermore, the region settings module is specifically used for:
[0138] Obtain the entry time when the first screen operation enters the first designated area, and the exit time when it leaves the first designated area;
[0139] Determine whether a second screen operation enters the first designated area within a second preset time period after the departure time;
[0140] If the second screen operation enters the first designated area, the second screen operation is determined to be part of the first screen operation, and the time when the second screen operation leaves the first designated area is updated to the departure time.
[0141] The duration of continuous operation is obtained by calculating the time difference between the departure time and the entry time.
[0142] In one embodiment, the locale setting module is specifically used for:
[0143] Obtain the departure time when the first screen operation leaves the first designated area;
[0144] Determine whether a third screen operation enters the first designated area within a third preset time period after the departure time;
[0145] If no third screen operation enters the first designated area, the display position of the first display window is restored to the first designated area.
[0146] In one embodiment, the screen operation information acquisition module is specifically used for:
[0147] Obtain the positions of multiple operation points of the first screen operation, and determine whether the position of each operation point falls within the first designated area;
[0148] If any of the operation points falls within the first designated area, then the screen operation position is determined to fall within the first designated area.
[0149] In one embodiment, the image compositing module is specifically used for:
[0150] Set the second specified area as the texture coordinates of the first display window;
[0151] The open graphics library is invoked to render the target sub-screen and the current display screen according to their corresponding texture coordinates, thereby obtaining the composite screen.
[0152] In one embodiment, the locale setting module is specifically used for:
[0153] The first display window is marked; the mark is used to indicate the display position of the first display window to the user.
[0154] Receive a position switching instruction for the first display window, and switch the display position of the first display window according to the position switching instruction.
[0155] Based on the same technical concept, embodiments of this application also provide a device for determining the position of a picture-in-picture in a composite video, see [link to relevant documentation]. Figure 8 ,include:
[0156] The screen operation information acquisition module is used to acquire the screen operation position of the first screen operation of the first display device in the current display screen, and to determine whether the screen operation position falls into the first designated area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen;
[0157] The region setting module is used to change the display position of the first display window from the first specified region to the second specified region when the screen operation position falls into the first specified region, so that the screen operation position does not fall into the second specified region.
[0158] In one embodiment, the locale setting module is specifically used for:
[0159] The duration of the first screen operation within the first specified area is obtained, and it is determined whether the duration of the continuous operation reaches a first preset duration.
[0160] If the continuous operation time reaches the first preset time, the display position of the first display window will be changed from the first designated area to the second designated area.
[0161] Furthermore, the region settings module is specifically used for:
[0162] Obtain the entry time when the first screen operation enters the first designated area, and the exit time when it leaves the first designated area;
[0163] Determine whether a second screen operation enters the first designated area within a second preset time period after the departure time;
[0164] If the second screen operation enters the first designated area, the second screen operation is determined to be part of the first screen operation, and the time when the second screen operation leaves the first designated area is updated to the departure time.
[0165] The duration of continuous operation is obtained by calculating the time difference between the departure time and the entry time.
[0166] In one embodiment, the locale setting module is specifically used for:
[0167] Obtain the departure time when the first screen operation leaves the first designated area;
[0168] Determine whether a third screen operation enters the first designated area within a third preset time period after the departure time;
[0169] If no third screen operation enters the first designated area, the display position of the first display window is restored to the first designated area.
[0170] In one embodiment, the screen operation information acquisition module is specifically used for:
[0171] Obtain the positions of multiple operation points of the first screen operation, and determine whether the position of each operation point falls within the first designated area;
[0172] If any of the operation points falls within the first designated area, then the screen operation position is determined to fall within the first designated area.
[0173] In one embodiment, the locale setting module is specifically used for:
[0174] The first display window is marked; the mark is used to indicate the display position of the first display window to the user.
[0175] Receive a position switching instruction for the first display window, and switch the display position of the first display window according to the position switching instruction.
[0176] It should be noted that the video compositing apparatus provided in the above embodiments, when executing the video compositing method, or the device for determining the picture-in-picture position of the composite video, when executing the method for determining the picture-in-picture position of the composite video, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device or electronic device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the video compositing apparatus and the video compositing method embodiments provided in the above embodiments belong to the same concept, and the device for determining the picture-in-picture position of the composite video and the method embodiment provided in the above embodiments belong to the same concept. For details of their implementation, please refer to the method embodiments, which will not be repeated here.
[0177] Based on the same technical concept, embodiments of this application also provide a smart interactive flat panel. See also Figure 9 The smart interactive flat panel includes a processor and a memory, the memory storing a computer program adapted to be loaded and executed by the processor to implement the method of any of the above embodiments.
[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of software products. The software product of the video synthesis method or the method for determining the picture-in-picture position of the synthesized video can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including storing several instructions to cause an electronic device to execute the methods of various embodiments or some parts of the embodiments.
[0179] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A video synthesis method, characterized in that, Applied to a first display device, the method includes: The screen operation position of the first screen operation of the first display device in the current display screen is obtained, and it is determined whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen; If the screen operation position falls within the first designated area, the display position of the first display window is changed from the first designated area to the second designated area so that the screen operation position does not fall within the second designated area. The target sub-screen is loaded into the first display window, and the next screen of the current display screen is loaded into the second display window to obtain the composite screen; wherein, the first display window is located on top of the second display window, and the second display window is used to display the next screen of the current display screen; the composite screen is transmitted from the first display device to the second display device for display.
2. The method as described in claim 1, characterized in that, Before changing the display position of the first display window from the first designated area to the second designated area when the screen operation position falls within the first designated area, the method further includes: The duration of the first screen operation within the first specified area is obtained, and it is determined whether the duration of the continuous operation reaches a first preset duration. If the continuous operation time reaches the first preset time, the display position of the first display window will be changed from the first designated area to the second designated area.
3. The method as described in claim 2, characterized in that, The step of obtaining the duration of the first screen operation within the first specified area specifically includes: Obtain the entry time when the first screen operation enters the first designated area, and the exit time when it leaves the first designated area; Determine whether a second screen operation enters the first designated area within a second preset time period after the departure time; If the second screen operation enters the first designated area, the second screen operation is determined to be part of the first screen operation, and the time when the second screen operation leaves the first designated area is updated to the departure time. The duration of continuous operation is obtained by calculating the time difference between the departure time and the entry time.
4. The method as described in claim 1, characterized in that, After changing the display position of the first display window from the first designated area to the second designated area, the method further includes: Obtain the departure time when the first screen operation leaves the first designated area; Determine whether a third screen operation enters the first designated area within a third preset time period after the departure time; If no third screen operation enters the first designated area, the display position of the first display window is restored to the first designated area.
5. The method as described in claim 1, characterized in that, The first screen operation is a multi-point operation; obtaining the screen operation position of the first screen operation on the first display device in the current display screen, and determining whether the screen operation position falls within the first designated area corresponding to the first display window, includes: Obtain the positions of multiple operation points of the first screen operation, and determine whether the position of each operation point falls within the first designated area; If any of the operation points falls within the first designated area, then the screen operation position is determined to fall within the first designated area.
6. The method as described in claim 1, characterized in that, The step of loading the target sub-screen into the first display window and loading the next screen of the current display screen into the second display window to obtain the composite screen includes: Set the second specified area as the texture coordinates of the first display window; The open graphics library is invoked to render the target sub-screen and the current display screen according to their corresponding texture coordinates, thereby obtaining the composite screen.
7. The method according to any one of claims 1-6, characterized in that, The method further includes: The first display window is marked; the mark is used to indicate the display position of the first display window to the user. Receive a position switching instruction for the first display window, and switch the display position of the first display window according to the position switching instruction.
8. A method for determining the position of a picture-in-picture in a composite video, characterized in that, Applied to a display device, the method includes: The screen operation position of the first screen operation of the display device in the current display screen is obtained, and it is determined whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen; If the screen operation position falls within the first designated area, the display position of the first display window is changed from the first designated area to the second designated area so that the screen operation position does not fall within the second designated area.
9. A video synthesis device, characterized in that, include: The screen operation information acquisition module is used to acquire the screen operation position of the first screen operation of the first display device in the current display screen, and to determine whether the screen operation position falls into the first designated area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen; The region setting module is used to change the display position of the first display window from the first designated region to the second designated region when the screen operation position falls into the first designated region, so that the screen operation position does not fall into the second designated region. The image compositing module is used to load the target sub-image into the first display window and load the next image of the current display image into the second display window to obtain the composite image; wherein, the first display window is located on top of the second display window, and the second display window is used to display the next image of the current display image; the composite image is transmitted from the first display device to the second display device for display.
10. A device for determining the position of a picture-in-picture in a composite video, characterized in that, include: The screen operation information acquisition module is used to acquire the screen operation position of the first screen operation of the display device in the current display screen, and to determine whether the screen operation position falls into the first specified area corresponding to the first display window; wherein, the first display window is used to display the target sub-screen in the composite screen; The region setting module is used to change the display position of the first display window from the first designated region to a second designated region when the screen operation position falls into the first designated region, so that the screen operation position does not fall into the second designated region.
11. A smart interactive flat panel, characterized in that, include: Processor and memory; The memory stores a computer program adapted to be loaded by the processor and executed by the video compositing method as described in any one of claims 1-7 or the method for determining the picture-in-picture position of the composite video as described in claim 8.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading and execution by a processor of the video compositing method as described in any one of claims 1-7 or the method for determining the picture-in-picture position of the composite video as described in claim 8.