Picture-in-picture video stream generation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a picture-in-picture video stream generation method and device, equipment and a storage medium, and the method comprises the steps: firstly, periodically extracting a frame of target image from a first video stream in a process of generating a picture-in-picture video stream, intercepting a sub-image corresponding to a candidate region from the target image, and obtaining a sub-image corresponding to the candidate region from the sub-image; and determining a target rectangular region according to the sub-image, and filling the target rectangular region in the first video stream with the second video stream in the current period to obtain a picture-in-picture video stream. According to the embodiment of the invention, the display position of the picture-in-picture can be automatically adjusted according to the image content displayed in the first video stream, a user does not need to manually adjust the display position of the picture-in-picture, the efficiency of adjusting the display position of the picture-in-picture is improved, and the user experience is improved. The technical problem that in the prior art, when a user manually adjusts the display position of the picture-in-picture, the adjustment efficiency is low is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular to a method, apparatus, device and storage medium for generating a picture-in-picture video stream. Background Art

[0002] Picture-in-picture is a video processing technology that can embed another video screen in one video screen. It is usually used in live broadcasts, TV programs, advertisements and other fields. For example, in a recording, interactive or live broadcast scenario, when a user watches a recorded screen or a live broadcast screen through the device's display screen, the first screen on the display screen interface is used to play the content explained by the speaker in full screen. In addition, a second screen for displaying the speaker's screen is also displayed within the first screen. At present, the position of the second screen is generally fixed. In some cases, the second screen may block the content explained by the speaker, requiring the speaker or backstage personnel to frequently manually adjust the position of the second screen, which is inefficient. Summary of the invention

[0003] The embodiments of the present invention provide a method, apparatus, device and storage medium for generating a picture-in-picture video stream. The embodiments of the present invention can automatically adjust the display position of the picture-in-picture, thereby solving the technical problem of low adjustment efficiency when a user manually adjusts the display position of the picture-in-picture in the prior art.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating a picture-in-picture video stream, including:

[0005] Receiving a first video stream and a second video stream in real time, and periodically extracting a frame of a target image from the first video stream, wherein the first video stream is a video stream generated by receiving a picture of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of a target object;

[0006] intercepting a sub-image corresponding to a candidate area from the target image, and determining a target rectangular area from the candidate area according to pixel distribution in the sub-image, wherein the candidate area is at least one area preset in the target image, and the aspect ratio of the target rectangular area is the same as the aspect ratio of the second video stream;

[0007] During an extraction period corresponding to the target image, the target rectangular area in the first video stream is filled with the second video stream to obtain a picture-in-picture video stream.

[0008] In a second aspect, an embodiment of the present invention provides a picture-in-picture video stream generating device, including:

[0009] An image extraction module is used to receive a first video stream and a second video stream in real time, and periodically extract a frame of a target image from the first video stream, wherein the first video stream is a video stream generated by receiving a picture of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of a target object;

[0010] A region determination module, configured to intercept a sub-image corresponding to a candidate region from the target image, and determine a target rectangular region from the candidate region according to pixel distribution in the sub-image, wherein the candidate region is at least one region preset in the target image, and the aspect ratio of the target rectangular region is the same as the aspect ratio of the second video stream;

[0011] The video stream filling module is used to fill the second video stream into the target rectangular area in the first video stream within the extraction period corresponding to the target image to obtain a picture-in-picture video stream.

[0012] In a third aspect, an embodiment of the present invention provides a picture-in-picture video stream generation device, the picture-in-picture video stream generation device comprising a processor and a memory;

[0013] The memory is used to store a computer program and transmit the computer program to the processor;

[0014] The processor is used to execute a picture-in-picture video stream generation method as described in the first aspect according to instructions in the computer program.

[0015] In a fourth aspect, an embodiment of the present invention provides a storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute a picture-in-picture video stream generation method as described in the first aspect.

[0016] As described above, in the process of generating a picture-in-picture video stream, the embodiment of the present invention first periodically extracts a frame of target image from the first video stream, cuts out a sub-image corresponding to the candidate area in the target image, determines the target rectangular area according to the sub-image, and fills the second video stream into the target rectangular area in the first video stream within this period to obtain the picture-in-picture video stream. The embodiment of the present invention can automatically adjust the display position of the picture-in-picture according to the image content displayed in the first video stream, without the need for the user to manually adjust the display position of the picture-in-picture, thereby improving the efficiency of adjusting the display position of the picture-in-picture and solving the technical problem of low adjustment efficiency when the user manually adjusts the display position of the picture-in-picture in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a picture-in-picture display interface of a display screen provided by an embodiment of the present invention.

[0018] Figure 2 A schematic diagram of a teaching live broadcast scenario provided in an embodiment of the present invention.

[0019] Figure 3 An overall flow chart of generating a live video stream in the prior art provided by an embodiment of the present invention.

[0020] Figure 4 A flowchart of a method for generating a picture-in-picture video stream provided by an embodiment of the present invention.

[0021] Figure 5 An overall flow chart of generating a picture-in-picture video stream provided by an embodiment of the present invention.

[0022] Figure 6 A flowchart of another method for generating a picture-in-picture video stream provided by an embodiment of the present invention.

[0023] Figure 7 A schematic diagram of a candidate region provided by an embodiment of the present invention.

[0024] Figure 8 A schematic diagram of determining an area where continuous pixels of the same color are located is provided in an embodiment of the present invention.

[0025] Fig. 9 A schematic diagram of determining a second area provided by an embodiment of the present invention.

[0026] Fig.10 A schematic structural diagram of a picture-in-picture video stream generating device provided by an embodiment of the present invention.

[0027] Fig.11 A structural schematic diagram of a picture-in-picture video stream generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following description and accompanying drawings fully illustrate the specific embodiments of the present application so that those skilled in the art can practice them. The examples represent possible variations only. Unless explicitly required, separate components and functions are optional, and the order of operation can vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. The scope of the embodiments of the present application includes the entire scope of the claims, and all available equivalents of the claims. In this article, each embodiment may be represented individually or generally by the term "invention", which is only for convenience, and if more than one invention is disclosed in fact, it is not intended to automatically limit the scope of the application to any single invention or inventive concept. In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. The various embodiments are described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. As for the structures, products, etc. disclosed in the embodiments, since they correspond to the parts disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0029] In the prior art, in a recording, interactive or live broadcast scenario, when a user watches a recorded screen or a live broadcast screen through a display screen of a device, the interface of the display screen generally displays a picture-in-picture interface, which includes two screens, namely a first screen and a second screen, such as Figure 1 As shown. The first screen is used to display the content explained by the lecturer, such as a courseware screen, a live broadcast screen, or a video screen, and the second screen is used to display the lecturer's own screen. For example, in the scenario of teaching live broadcast, Figure 2As shown, the teacher can play the courseware through the interactive board and explain the content of the courseware. At this time, the video recorder can collect the audio and video data of the interactive board as the audio and video data to be played on the first screen, and collect the teacher's image and sound as the audio and video data of the second screen through the camera and microphone. Specifically, after collecting the audio and video data of the interactive board and the audio and video data of the teacher, the video recorder decodes the audio data and video data of the interactive board, decodes the audio data and video data of the teacher, and then mixes the decoded audio data of the board and the decoded audio data of the teacher and then encodes them to generate ACC (Advanced Audio Coding) audio data. At the same time, the video recorder fills the decoded teacher's video data into the decoded interactive board's video data according to the position of the second picture in the first picture of the picture-in-picture, and then encodes the filled video data to generate H264 (highly compressed digital video codec standard) video data. Finally, the live video stream in the teaching live broadcast scenario is generated according to the ACC audio data and H264 video data, and the live video stream is sent to the network for dissemination. The specific process is as follows Figure 3 As shown. After that, students can obtain the live video stream from the network through mobile phones and other devices and play it, and then they can watch the screen displayed on the interactive board and the teacher's screen in real time in a picture-in-picture manner.

[0030] However, in some scenarios, the second screen may block the content displayed on the first screen, requiring the video recorder to manually reduce, move or hide the second screen during the live broadcast. When the content of the first screen changes frequently, the video recorder also needs to frequently adjust the position of the second screen, and the adjustment efficiency is very low.

[0031] Based on this, in order to solve the above problems, an embodiment of the present invention provides a method for generating a picture-in-picture video stream, such as Figure 4 As shown, Figure 4 A flowchart of a method for generating a picture-in-picture video stream provided by an embodiment of the present invention. The method for generating a picture-in-picture video stream provided by an embodiment of the present invention can be executed by a picture-in-picture video stream generating device, which can be implemented by software and / or hardware. The picture-in-picture video stream generating device can be composed of two or more physical entities, or can be composed of one physical entity. For example, the picture-in-picture video stream generating device can be a computer, a host computer, or a tablet. The method includes the following steps:

[0032] Step S101, receiving a first video stream and a second video stream in real time, and periodically extracting a frame of target image from the first video stream, wherein the first video stream is a video stream generated by receiving the screen of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of the target object.

[0033] In this embodiment, the picture-in-picture video stream generating device needs to receive the first video stream and the second video stream in real time, wherein the video stream refers to video data that is streamed and played in a streaming media file format, and the streaming mode is to compress multimedia files such as video data into compressed packages, so that the video data can be played in real time without downloading and waiting. In addition, in this embodiment, the sources of the first video stream and the second video stream are different, wherein the first video stream is a video stream generated by receiving the screen of the terminal device in real time, for example, the terminal device can be an interactive large board, a computer or a tablet with a screen display capability. The second video stream is a video stream generated by capturing an image of a target object, wherein the target object is a pre-set object, for example, the target object can be a host, a teacher or an anchor, etc., and the target object can be set according to actual needs. In addition, the second video stream can be collected by a device such as a camera, and the collection method of the second video stream is not specifically limited in this embodiment.

[0034] In addition, in this embodiment, after receiving the first video stream in real time, the picture-in-picture video stream generating device needs to periodically extract a frame of image from the first video stream as a target image, and the duration of the period can also be set according to actual needs. For example, a period can be set to the duration of the first video stream transmitting 5 frames of images, that is, after each 5 frames of images of the first video stream are received, a target image is extracted from the first video stream.

[0035] Step S102, capture a sub-image corresponding to the candidate area from the target image, and confirm a target rectangular area from the candidate area based on the pixel distribution in the sub-image, where the candidate area is at least one area preset in the target image, and the aspect ratio of the target rectangular area is the same as the aspect ratio of the second video stream.

[0036] After a target image is extracted from the first video stream in each cycle, a sub-image needs to be further intercepted from a candidate area of ​​the target image, wherein the candidate area is at least one area pre-set in the target image, and the number and position of the candidate areas can be set according to actual needs. For example, the number of candidate areas can be set to four, and the four candidate areas are respectively located at the four corners of the target image.

[0037] After the sub-image is cut out, the target rectangular frame can be determined in the candidate area of ​​the target image based on the pixel distribution in the sub-image, wherein the pixel distribution refers to the color distribution of each pixel in the sub-image; the target rectangular frame refers to a rectangular frame that meets the preset conditions and has the same aspect ratio as the video screen of the second video stream, and the aspect ratio refers to the ratio of length to width. Exemplarily, the preset condition can be set to the area where the image background with the largest area is located in the sub-image, and when the target rectangular area is subsequently determined from the candidate area, the area where the image background with the largest area is located can be determined in the sub-image based on the distribution of pixels in the sub-image, and a target rectangular area with the same aspect ratio as the aspect ratio of the second video stream can be determined in the area where the image background with the largest area is located.

[0038] Step S103: within the extraction cycle corresponding to the target image, fill the target rectangular area in the first video stream with the second video stream to obtain a picture-in-picture video stream.

[0039] After determining the target rectangular area, the picture-in-picture video stream generation device can fill the second video stream into the target rectangular area of ​​the first video stream within the period of extracting the target image, thereby generating a picture-in-picture video stream, wherein the picture-in-picture video stream refers to a video stream including a picture-in-picture. After receiving the picture-in-picture video stream, other devices can decode the picture-in-picture video stream to display the picture-in-picture. In this embodiment, since the image background of the first video stream is displayed in the target rectangular area, when the second video stream is displayed in the target rectangular area, the second video stream will not block the area in the first video stream that displays important content. When the next cycle arrives, the picture-in-picture video stream generation device can again extract the target image from the first video stream, and re-update the position of the target rectangular area according to the target image, so that the position of the target rectangular area can change according to the image content in the first video stream, without the user manually adjusting the position of the picture-in-picture. In addition, it can be understood that the audio data in the first video stream and the audio data in the second video stream are processed in the same manner as in the prior art, that is, the audio data in the first video stream and the audio data in the second video stream are mixed and then encoded to generate ACC audio data, which will not be described in detail in this embodiment. In one embodiment, the specific process of obtaining the picture-in-picture video stream is as follows: Figure 5 shown.

[0040] As described above, in the process of generating a picture-in-picture video stream, the embodiment of the present invention first periodically extracts a frame of target image from the first video stream, cuts out a sub-image corresponding to the candidate area in the target image, determines the target rectangular area according to the sub-image, and fills the second video stream into the target rectangular area in the first video stream within this period to obtain the picture-in-picture video stream. The embodiment of the present invention can automatically adjust the display position of the picture-in-picture according to the image content displayed in the first video stream, without the need for the user to manually adjust the display position of the picture-in-picture, thereby improving the efficiency of adjusting the display position of the picture-in-picture and solving the technical problem of low adjustment efficiency when the user manually adjusts the display position of the picture-in-picture in the prior art.

[0041] like Figure 6 As shown, Figure 6 A schematic diagram of a flow chart of another method for generating a picture-in-picture video stream provided by an embodiment of the present invention, Figure 6 The picture-in-picture video stream generation method shown is a specific embodiment of the above-mentioned picture-in-picture video stream generation method. The picture-in-picture video stream generation method provided in this embodiment includes:

[0042] Step 201, receiving a first video stream and a second video stream in real time, and periodically extracting a frame of target image from the first video stream, wherein the first video stream is a video stream generated by receiving the screen of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of the target object.

[0043] Step 202: Determine image data in a target image that is within a candidate region; the candidate region is at least one region preset in the target image.

[0044] In this embodiment, after extracting the target image from the first video stream, it is first necessary to determine the candidate area in the target image, and determine the image data in the target image that is located in the candidate area. In one embodiment, the target image includes four candidate areas, and a corner point in each of the four candidate areas coincides with four corner points of the target image, where a corner point is the intersection of two lines. Figure 7 As shown, Figure 7 The four candidate areas in the target image are rectangular areas, namely S1, S2, S3 and S4, where the upper left corner of S1 coincides with the upper left corner of the target image, the upper right corner of S2 coincides with the upper right corner of the target image, the lower left corner of S3 coincides with the lower left corner of the target image, and the lower right corner of S4 coincides with the lower right corner of the target image. After acquiring the target image, the image data within S1, S2, S3 and S4 can be determined.

[0045] Step 203: extract image data in the candidate area from the target image to obtain a sub-image corresponding to the candidate area.

[0046] After the image data in the candidate area is determined, the image data in the candidate area can be cut out from the target image to obtain a sub-image corresponding to the candidate area. Figure 7 For the candidate area shown, it is necessary to extract the image data in S1, S2, S3 and S4 to obtain four sub-images.

[0047] Step 204: Convert each sub-image into an HSV image.

[0048] After the sub-image is obtained, since the target image extracted from the first video stream is an RGB (red, green, blue, composed of red, green, and blue grayscale data) image, the sub-image is also an RGB image. Since the RGB color space is a color space with poor uniformity, it is difficult to determine a relatively accurate three-component numerical value to represent a certain color. Therefore, the RGB color space is suitable for display systems, but not for image processing. Based on this, in order to facilitate subsequent image processing, the sub-image needs to be further converted into an HSV (Hue, Saturation, Value) image in this embodiment. HSV is a color space created based on the intuitive characteristics of color, also known as a hexagonal pyramid model. The HSV color model refers to a subset of visible light in the H, S, V three-dimensional color space. It contains all colors in a certain color domain. HSV is a relatively intuitive color model and is widely used in computer graphics. It can be understood that the process of converting an RGB image to an HSV image can refer to the prior art and will not be repeated in this embodiment.

[0049] Step 205: extract the pixel color of each HSV image, and confirm the target rectangular area from the candidate area according to the pixel color. The aspect ratio of the target rectangular area is the same as the aspect ratio of the second video stream.

[0050] After converting the sub-image into an HSV image, the pixel color of each HSV image can be extracted, where a pixel refers to a minimum unit in an image represented by a digital sequence, and a pixel color refers to the color corresponding to the pixel. The pixel color of an HSV image can be obtained by extracting the hue of the HSV image. Afterwards, the target rectangular area can be determined in the HSV image according to the distribution of the pixel colors, thereby obtaining the target rectangular area in the candidate area.

[0051] Based on the above embodiment, in step 205, determining the target rectangular area from the candidate areas according to the pixel colors includes:

[0052] Step 2051: According to the pixel color of each HSV image, determine in all HSV images a first region where the same pixel color is continuous and has the largest area.

[0053] In one embodiment, when determining the target rectangular area from the candidate area, the first area where the same pixel color is continuous and has the largest area proportion can be determined in all HSV images based on the pixel color of each HSV image. The continuous same pixel color refers to the colors of two adjacent pixels being the same. It should be noted that the image data of the target image generally includes an image background and an image subject. The image background is used to highlight the image subject and generally does not carry valid information. The image background generally uses a solid color pattern. Therefore, by determining the continuous same pixel color, the area where the image background is located can be determined in the HSV image. Exemplarily, for example, Figure 7 The target image shown in FIG. 1 is used to determine the areas where the continuous pixels of the same color are located in S1, S2, S3 and S4. Figure 8 shown.

[0054] After determining the continuous pixel colors of the same type in the HSV image, the first region with the largest continuous pixel colors can be determined in all HSV images, so that the image of the second video stream can be displayed as large as possible subsequently.

[0055] Step 2052: Determine in the first region a second region having the same aspect ratio as that of the second video stream and having the largest area.

[0056] After the first region is determined in all HSV images, since the first region may be a completely irregular region, in order to fully display the image of the second video stream later, it is necessary to further determine the second region with the same aspect ratio as the aspect ratio of the second video stream and the largest area in the first region. For example, if the aspect ratio of the second video stream is 16:9, then in this embodiment, it is necessary to determine the second region with the aspect ratio of 16:9 and the largest area in the first region. For example, Fig. 9 As shown, the first area determined in the target image according to the HSV image is D1, and then a rectangular area with an aspect ratio of 16:9 and the largest area is further determined in D1 as the second area.

[0057] Step 2053: determine the second area as the target rectangular area.

[0058] Finally, the second area can be determined as the target rectangular area.

[0059] The above is the specific process of determining the target rectangular area in the target image.

[0060] Step 206: within the extraction period corresponding to the target image, convert the second video stream according to the area of ​​the target rectangular area so that the converted second video stream fills all areas of the target rectangular area.

[0061] After determining the target rectangular area, the second video stream can be converted according to the area of ​​the target rectangular area within the period of extracting the target image, that is, the size of the second video stream can be enlarged or reduced according to the area of ​​the target rectangular area, so that when the converted second video stream is displayed, the picture of the second video stream can fill all areas of the target rectangular area.

[0062] Step 207: Fill the converted second video stream into the target rectangular area in the first video stream to obtain a picture-in-picture video stream.

[0063] Finally, each frame of the second video stream after the size reduction can be filled into the target rectangular area of ​​the image corresponding to the first video stream to obtain the picture-in-picture video stream. In the current cycle, when the user obtains the picture-in-picture video stream and plays it, the picture corresponding to the second video stream can be displayed in the target rectangular area of ​​the picture displayed by the first video stream.

[0064] On the basis of the above embodiment, after confirming the target rectangular area from the candidate area according to the pixel color and before filling the second video stream into the target rectangular area in the first video stream, the method further includes:

[0065] Determine the area of ​​the target rectangle.

[0066] When the area is smaller than the preset area, the second video stream is discarded.

[0067] In one embodiment, after determining the target rectangular area from the candidate area of ​​the target image, it is first necessary to determine the area of ​​the target rectangular area. When the area of ​​the target rectangular area is less than the preset area, the second video stream is discarded. The preset area is a pre-set area, and the size of the preset area can be set according to actual needs. In one embodiment, the user can set the preset area to the minimum acceptable area. If the area of ​​the target rectangular area is less than the preset area, it means that the area of ​​the target rectangular area is too small. At this time, when the screen of the second video stream is displayed in the preset rectangular area, the user will not be able to see the screen content clearly due to the small area of ​​the screen, that is, it is meaningless to display the second video stream at this time. Therefore, when the area of ​​the target rectangular area is less than the preset area, the second video stream can be discarded, that is, the second video stream is not filled into the target rectangular area of ​​the first video stream, and only the first video stream is transmitted.

[0068] As described above, in the process of generating a picture-in-picture video stream, the embodiment of the present invention first periodically extracts a frame of the target image from the first video stream, cuts out a sub-image corresponding to the candidate area from the target image, and then converts the sub-image into an HSV image, and in the HSV image, determines the first area where the same pixel color is continuous and has the largest area, and determines the target rectangular area in the first area. The embodiment of the present invention determines the area where the image background is located in the HSV image by determining the area where the same pixel color is continuous, so that the picture corresponding to the second video stream can be displayed in the image background of the first video stream, thereby avoiding blocking the important content displayed in the first video stream. The embodiment of the present invention can automatically adjust the display position of the picture-in-picture according to the image content displayed in the first video stream, without the user manually adjusting the display position of the picture-in-picture, thereby improving the efficiency of adjusting the display position of the picture-in-picture, and solving the technical problem of low adjustment efficiency when the user manually adjusts the display position of the picture-in-picture in the prior art.

[0069] like Fig.10 As shown, Fig.10 A schematic diagram of the structure of a picture-in-picture video stream generating device provided by an embodiment of the present invention is shown in FIG. Fig.10 As shown, the embodiment of the present invention provides a picture-in-picture video stream generating device including:

[0070] The image extraction module 301 is used to receive a first video stream and a second video stream in real time, and periodically extract a frame of target image from the first video stream, wherein the first video stream is a video stream generated by receiving the screen of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of the target object;

[0071] The region determination module 302 is used to intercept a sub-image corresponding to the candidate region from the target image, and determine a target rectangular region from the candidate region according to pixel distribution in the sub-image, where the candidate region is at least one region preset in the target image, and the aspect ratio of the target rectangular region is the same as the aspect ratio of the second video stream;

[0072] The video stream filling module 303 is used to fill the target rectangular area in the first video stream with the second video stream within the extraction period corresponding to the target image to obtain a picture-in-picture video stream.

[0073] Based on the above embodiment, the region determination module 302 includes:

[0074] An image data determination submodule, used to determine image data located in a candidate area in a target image;

[0075] The sub-image capture sub-module is used to capture the image data in the candidate area from the target image to obtain a sub-image corresponding to the candidate area.

[0076] Based on the above embodiment, the region determination module 302 includes:

[0077] HSV conversion submodule, used to convert each sub-image into an HSV image;

[0078] The target rectangular area determination submodule is used to extract the pixel color of each HSV image and confirm the target rectangular area from the candidate area according to the pixel color.

[0079] Based on the above embodiment, the target rectangular area determination submodule includes:

[0080] A first region determining unit is used to determine, according to the pixel color of each HSV image, a first region where the same pixel color is continuous and has the largest area in all HSV images;

[0081] A second region determining unit, configured to determine, in the first region, a second region having the same aspect ratio as that of the second video stream and having the largest area;

[0082] The target rectangular area determining unit is used to determine the second area as the target rectangular area.

[0083] Based on the above embodiment, it also includes:

[0084] an area determination module, for determining the area of ​​the target rectangular area after confirming the target rectangular area from the candidate area according to the pixel color and before filling the second video stream into the target rectangular area in the first video stream;

[0085] The video stream discarding module is used to discard the second video stream when the area is smaller than a preset area.

[0086] On the basis of the above embodiment, the target image includes four candidate areas, and one corner point in the four candidate areas coincides with four corner points of the target image respectively.

[0087] Based on the above embodiment, the video stream filling module 303 includes:

[0088] The video stream conversion submodule is used to convert the second video stream according to the area of ​​the target rectangular area so that the converted second video stream fills all areas of the target rectangular area;

[0089] The video stream filling submodule is used to fill the converted second video stream into the target rectangular area in the first video stream to obtain a picture-in-picture video stream.

[0090] The picture-in-picture video stream generating apparatus provided in the embodiment of the present invention is included in a picture-in-picture video stream generating device and can be used to execute the picture-in-picture video stream generating method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0091] It is worth noting that in the embodiment of the above-mentioned picture-in-picture video stream generating device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0092] This embodiment also provides a picture-in-picture video stream generation device, such as Fig.11 As shown, the picture-in-picture video stream generating device 40 includes a processor 400 and a memory 401;

[0093] The memory 401 is used to store the computer program 402 and transmit the computer program 402 to the processor 400;

[0094] The processor 400 is used to execute the steps in the above-mentioned embodiment of a method for generating a picture-in-picture video stream according to the instructions in the computer program 402 .

[0095] Exemplarily, the computer program 402 may be divided into one or more modules / units, one or more modules / units are stored in the memory 401, and are executed by the processor 400 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 402 in the picture-in-picture video stream generating device 40.

[0096] The picture-in-picture video stream generating device 40 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The picture-in-picture video stream generating device 40 may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art will appreciate that Fig.11 It is only an example of the picture-in-picture video stream generating device 40 and does not constitute a limitation of the picture-in-picture video stream generating device 40. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the picture-in-picture video stream generating device 40 may also include input and output devices, network access devices, buses, etc.

[0097] The processor 400 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0098] The memory 401 may be an internal storage unit of the picture-in-picture video stream generating device 40, such as a hard disk or memory of the picture-in-picture video stream generating device 40. The memory 401 may also be an external storage device of the picture-in-picture video stream generating device 40, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the picture-in-picture video stream generating device 40. Further, the memory 401 may also include both the internal storage unit of the picture-in-picture video stream generating device 40 and an external storage device. The memory 401 is used to store computer programs and other programs and data required by the picture-in-picture video stream generating device 40. The memory 401 may also be used to temporarily store data that has been output or is to be output.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store computer programs.

[0104] An embodiment of the present invention further provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute a method for generating a picture-in-picture video stream. The method includes the following steps:

[0105] Receive a first video stream and a second video stream in real time, and periodically extract a frame of target image from the first video stream, wherein the first video stream is a video stream generated by receiving the screen of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of the target object;

[0106] Intercepting a sub-image corresponding to the candidate area from the target image, and determining a target rectangular area from the candidate area according to pixel distribution in the sub-image, wherein the candidate area is at least one area preset in the target image, and the aspect ratio of the target rectangular area is the same as the aspect ratio of the second video stream;

[0107] In an extraction cycle corresponding to the target image, the second video stream is filled into the target rectangular area in the first video stream to obtain a picture-in-picture video stream.

[0108] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the embodiments of the present invention are not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of the present invention. Therefore, although the embodiments of the present invention are described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the embodiments of the present invention, and the scope of the embodiments of the present invention is determined by the scope of the appended claims.

Claims

1. A method for generating a picture-in-picture video stream, characterized in that: include: Receiving a first video stream and a second video stream in real time, and periodically extracting a frame of a target image from the first video stream, wherein the first video stream is a video stream generated by receiving a picture of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of a target object; intercepting a sub-image corresponding to a candidate area from the target image, and determining a target rectangular area from the candidate area according to pixel distribution in the sub-image, wherein the candidate area is at least one area preset in the target image, and the aspect ratio of the target rectangular area is the same as the aspect ratio of the second video stream; During an extraction period corresponding to the target image, the target rectangular area in the first video stream is filled with the second video stream to obtain a picture-in-picture video stream.

2. A method for generating a picture-in-picture video stream according to claim 1, characterized in that: The step of intercepting a sub-image corresponding to the candidate area from the target image comprises: Determine image data in the target image that is located in the candidate area; Image data within the candidate area is cut out from the target image to obtain a sub-image corresponding to the candidate area.

3. The method for generating a picture-in-picture video stream according to claim 1, characterized in that: The step of confirming a target rectangular area from the candidate area according to the pixel distribution in the sub-image comprises: Convert each of the sub-images into an HSV image; The pixel color of each of the HSV images is extracted, and the target rectangular area is confirmed from the candidate areas according to the pixel color.

4. The method for generating a picture-in-picture video stream according to claim 3, characterized in that: The step of determining a target rectangular area from the candidate areas according to the pixel colors includes: According to the pixel color of each of the HSV images, determine in all the HSV images a first region where the same pixel color is continuous and has the largest area proportion; Determine in the first area a second area having the same aspect ratio as that of the second video stream and having the largest area; The second area is determined as the target rectangular area.

5. The method for generating a picture-in-picture video stream according to claim 1, characterized in that: After determining the target rectangular area from the candidate area according to the pixel color and before filling the second video stream into the target rectangular area in the first video stream, the method further includes: Determine the area of ​​the target rectangular area; When the area is smaller than a preset area, the second video stream is discarded.

6. The method for generating a picture-in-picture video stream according to claim 1, characterized in that: The target image includes four candidate areas, and one corner point in each of the four candidate areas coincides with four corner points of the target image respectively.

7. The method for generating a picture-in-picture video stream according to claim 1, characterized in that: The step of filling the target rectangular area in the first video stream with the second video stream to obtain a picture-in-picture video stream includes: Converting the second video stream according to the area of ​​the target rectangular area so that the converted second video stream fills all areas of the target rectangular area; The converted second video stream is filled into the target rectangular area in the first video stream to obtain a picture-in-picture video stream.

8. A picture-in-picture video stream generating device, characterized in that: include: An image extraction module is used to receive a first video stream and a second video stream in real time, and periodically extract a frame of a target image from the first video stream, wherein the first video stream is a video stream generated by receiving a picture of a terminal device in real time, and the second video stream is a video stream generated by capturing an image of a target object; A region determination module, configured to intercept a sub-image corresponding to a candidate region from the target image, and determine a target rectangular region from the candidate region according to pixel distribution in the sub-image, wherein the candidate region is at least one region preset in the target image, and the aspect ratio of the target rectangular region is the same as the aspect ratio of the second video stream; The video stream filling module is used to fill the second video stream into the target rectangular area in the first video stream within the extraction period corresponding to the target image to obtain a picture-in-picture video stream.

9. A picture-in-picture video stream generating device, characterized in that: The picture-in-picture video stream generating device comprises a processor and a memory; The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute a picture-in-picture video stream generation method according to any one of claims 1-7 according to the instructions in the computer program.

10. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to execute a method for generating a picture-in-picture video stream according to any one of claims 1 to 7.

Images