Live picture output method and device, computer device and readable storage medium

Abstract

The application discloses a live picture output method and device, computer equipment and a readable storage medium, relates to the visual processing technical field, and based on double cameras, not only panoramic pictures are collected, close-up shooting is carried out, the content displayed is more various, and users can understand live content more carefully. The method comprises the following steps: in response to a live start instruction, real-time image collection is carried out on a current live scene based on a main camera and an auxiliary camera, a main video stream collected by the main camera is acquired; a to-be-close-up object is determined, image recognition is carried out on the main video stream, object frame coordinates of the object frame of the to-be-close-up object on the main video stream are determined; the shooting angle of the auxiliary camera is adjusted according to the object frame coordinates, the auxiliary camera is controlled to face the to-be-close-up object and image collection is carried out on the to-be-close-up object, an auxiliary video stream is obtained; a picture-in-picture close-up area is determined in the image picture of the main video stream, the image picture of the auxiliary video stream is projected to the picture-in-picture close-up area, and a live picture is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual processing, in particular to a live picture output method and device, computer equipment and readable storage medium. BACKGROUND

[0002] Today's network live technology is becoming mature with the extensive and in-depth development of self-media, and the network live platform has become a new social media. Network live technology mainly includes real-time live games, movies or TV series, etc. At present, there are more than one hundred network live platforms, among which there are dozens of most famous ones, which can be described as a hundred flowers blooming.

[0003] In the related art, in some shopping live scenes, the host on the host side uses a terminal device such as a mobile phone or a computer to start broadcasting to introduce products. The terminal device uploads the video stream formed by the live broadcast to the live server, and the live server inputs the video stream to each audience side for playing, so that the user on the audience side can see the live picture. However, the applicant realizes that the live picture output by the host side is usually a two-dimensional, single-angle picture, the content displayed by the live picture is single, and the amount of information provided is limited, which makes it difficult for the user watching the live broadcast to understand the product in detail through the live picture output by the host side. SUMMARY

[0004] Therefore, the present application provides a live picture output method and device, computer equipment and readable storage medium, which mainly aims to solve the problem that the live picture output by the host side is usually a two-dimensional, single-angle picture, the content displayed by the live picture is single, and the amount of information provided is limited, which makes it difficult for the user watching the live broadcast to understand the product in detail through the live picture output by the host side.

[0005] According to a first aspect of the present application, a live picture output method is provided, which comprises:

[0006] In response to a live start instruction, real-time image acquisition is performed on a current live scene based on a main camera and a sub-camera, and a main video stream collected by the main camera is obtained;

[0007] A to-be-close-up object is determined, image recognition is performed on the main video stream, and object frame coordinates of the object frame of the to-be-close-up object on the main video stream are determined;

[0008] According to the object frame coordinates, the shooting angle of the sub-camera is adjusted, the sub-camera is controlled to face the to-be-close-up object and perform image acquisition on the to-be-close-up object, and a sub-video stream is obtained;

[0009] A picture-in-picture close-up area is determined in the image picture of the main video stream, the image picture of the sub-video stream is projected to the picture-in-picture close-up area, and a live picture is obtained.

[0010] Optionally, the determining the object to be close-up and the image recognition on the main video stream to determine the object frame coordinates of the object frame of the object to be close-up on the main video stream comprises:

[0011] acquiring preset live broadcast parameters, and specifying the object to be close-up at a live broadcast time specified by the live broadcast parameters as the object to be close-up, the object to be close-up being any one of an anchor or a commodity;

[0012] when the object to be close-up is the anchor, performing human body recognition on the anchor to obtain the object frame, and reading the coordinates of the object frame on the main video stream as the object frame coordinates;

[0013] when the object to be close-up is the commodity, constructing the object frame according to the contour of the commodity, and reading the coordinates of the object frame on the main video stream as the object frame coordinates.

[0014] Optionally, the human body recognition on the anchor to obtain the object frame comprises:

[0015] performing down-sampling processing on the main video stream to obtain the main video stream after down-sampling processing;

[0016] performing image recognition on the processed main video stream by using a human body detection model to determine multiple human body key points and the object frame of the object to be close-up in the main video stream.

[0017] Optionally, the constructing the object frame according to the contour of the commodity comprises:

[0018] performing down-sampling processing on the main video stream to obtain the main video stream after down-sampling processing;

[0019] recognizing a commodity in the processed main video stream, taking the commodity as the object to be close-up, and the commodity being pasted with a commodity identification code or being consistent with a commodity picture uploaded by the anchor;

[0020] recognizing the contour of the object to be close-up, and constructing the object frame of the object to be close-up according to the contour.

[0021] Optionally, the adjusting the shooting angle of the secondary camera according to the object frame coordinates, controlling the secondary camera to face the object to be close-up and performing image acquisition on the object to be close-up to obtain a secondary video stream comprises:

[0022] controlling the secondary camera to rotate in the direction of the object to be close-up according to the object frame coordinates, and adjusting the shooting angle of the secondary camera;

[0023] When the sub-camera is controlled to rotate, the sub-camera is controlled to detect an object identification point by using an object detection model, the object identification point being a human head or a center point of a commodity;

[0024] When the object identification point is detected based on the sub-camera, a coordinate of the object identification point in an image currently captured by the sub-camera is continuously obtained;

[0025] Based on the coordinate of the object identification point, a position offset between the object frame and a camera visual center of the sub-camera is continuously calculated;

[0026] According to the position offset, the sub-camera is continuously controlled to rotate until the position offset is less than an offset threshold, and the rotation is stopped, so that the sub-camera faces the object to be close-up;

[0027] The sub-camera is controlled to capture an image of a direction currently faced by the sub-camera, and an auxiliary video stream is obtained.

[0028] Optionally, the continuously calculating, based on the coordinate of the object identification point, a position offset between the object frame and a camera visual center of the sub-camera comprises:

[0029] A visual center coordinate of the camera visual center is read, and an Euclidean distance between the coordinate of the object identification point and the visual center coordinate is calculated;

[0030] A camera visual frame is constructed with the camera visual center as a center point, and an intersection-over-union between the camera visual frame and the object frame is calculated;

[0031] A first weight value corresponding to the Euclidean distance and a second weight value corresponding to the intersection-over-union are queried, and the Euclidean distance and the intersection-over-union are weighted calculated by using the first weight value and the second weight value, and a calculation result obtained is taken as the position offset between the object frame and the camera visual center of the sub-camera.

[0032] Optionally, the determining a picture-in-picture close-up area in an image frame of the main video stream, projecting an image frame of the auxiliary video stream to the picture-in-picture close-up area, and obtaining a live picture comprises:

[0033] It is detected whether there is a projectable area in the image frame of the main video stream;

[0034] When it is detected that the projectable area exists in the image frame of the main video stream, a picture-in-picture area is formed by using the projectable area, the auxiliary video stream is projected in the picture-in-picture close-up area in the main video stream for playing, and the live picture is obtained;

[0035] When it is detected that the projection area does not exist in the image frame of the main video stream, a target sub-window is selected as the picture-in-picture area in the image frame of the main video stream, and the auxiliary video stream is projected and played in the picture-in-picture close-up area in the main video stream, to obtain the live picture.

[0036] Optionally, the forming of the picture-in-picture area by using the projection area comprises:

[0037] The image segmentation model is used to identify a polygon area of a preset color in the image frame of the main video stream, the polygon area is taken as the projection area, an image mask with an area consistent with that of the projection area is generated, and the image mask is superimposed on the projection area to form a picture-in-picture close-up area; or,

[0038] An augmented reality (AR) identification code is identified in the image frame of the main video stream, an area covered by the AR identification code is taken as the projection area, an image mask with an area consistent with that of the projection area is generated, and the image mask is superimposed on the projection area to form a picture-in-picture close-up area.

[0039] Optionally, the selecting of the target sub-window as the picture-in-picture area in the image frame of the main video stream comprises:

[0040] Anchor coordinates of an anchor and product coordinates of a product are read in the main video stream.

[0041] According to the anchor coordinates and the product coordinates, a to-be-displayed area including the anchor and the product is divided in the main video stream.

[0042] A specified number of sub-windows are set on the image frame of the main video stream, and an intersection-over-union ratio between each sub-window and the to-be-displayed area is calculated.

[0043] A target sub-window with the smallest intersection-over-union ratio and smaller than an intersection-over-union ratio threshold is selected from the specified number of sub-windows, and the target sub-window is taken as the picture-in-picture area.

[0044] Optionally, the method further comprises:

[0045] When the to-be-close-up object is the anchor, a product with a product identification code pasted thereon is identified in the main video stream, a combined feature vector is generated, and the combined feature vector is input into a preset pose classifier, the combined feature vector being used to represent a positional relationship between a hand key point of the anchor and the product.

[0046] If the recognition result output by the preset posture classifier indicates that the hand of the host is in contact with the commodity, a to-be-displayed region including the host and the commodity is divided in the host video stream according to the host coordinates and the commodity coordinates;

[0047] The host video stream is cropped and scaled according to the to-be-displayed region, and a processed host video stream is obtained.

[0048] Optionally, the generating the combined feature vector comprises:

[0049] A plurality of human body key points of the host are recognized, and a commodity identification code coordinate of the commodity identification code in the host video stream is obtained;

[0050] A human body coordinate system is established with the top-left vertex of the object frame as the origin, the commodity identification code coordinate is converted, the image coordinate system in which the commodity identification code coordinate is located is transformed to the human body coordinate system, and the converted commodity identification code coordinate is normalized to obtain a first normalized coordinate;

[0051] A plurality of key point coordinates corresponding to the plurality of human body key points are converted, the image coordinate system in which the plurality of key point coordinates are located is transformed to the human body coordinate system, and the converted plurality of key point coordinates are respectively normalized to obtain a plurality of second normalized coordinates;

[0052] The first normalized coordinate and the plurality of second normalized coordinates are combined to generate a feature vector, and the feature vector is taken as a combined feature vector.

[0053] Optionally, the method further comprises:

[0054] The live picture is output to a viewer end;

[0055] When a play adjustment instruction of the viewer end is received, to-be-played content indicated by the play adjustment instruction is determined;

[0056] The live picture is adjusted according to the to-be-played content, and the adjusted live picture is output to the viewer end.

[0057] Optionally, the adjusting the live picture according to the to-be-played content and outputting the adjusted live picture to the viewer end comprises:

[0058] When the to-be-played content is a host camera image, the host video stream is taken as the live picture, and the live picture is output to the viewer end, so that the viewer end plays the image picture of the host video stream in full screen;

[0059] When the content to be played is a close-up image, the auxiliary video stream is taken as the live picture, and the live picture is output to the audience terminal to make the audience terminal play the image picture of the auxiliary video stream in full screen;

[0060] When the content to be played is a split-screen image, a split-screen picture is created, a first projection area and a second projection area are divided on the split-screen picture, the main video stream is projected to the first projection area, the auxiliary video stream is projected to the second projection area, and the projected split-screen picture is taken as the live picture, which is output to the audience terminal to make the audience terminal play the projected split-screen picture in full screen.

[0061] According to the second aspect of the present application, an output device of a live picture is provided, which comprises:

[0062] An acquisition module is configured to, in response to a live start instruction, acquire a main video stream collected by a main camera based on real-time image collection of a current live scene by the main camera and a sub-camera.

[0063] An intelligent identification module is configured to determine an object to be close-up and determine object frame coordinates of an object frame of the object to be close-up on the main video stream through image identification of the main video stream.

[0064] An adjustment module is configured to adjust a shooting angle of the sub-camera according to the object frame coordinates, control the sub-camera to face the object to be close-up and collect images of the object to be close-up, and obtain an auxiliary video stream.

[0065] A director module is configured to determine a picture-in-picture close-up area in an image picture of the main video stream, project an image picture of the auxiliary video stream to the picture-in-picture close-up area, and obtain a live picture.

[0066] Optionally, the intelligent identification module is configured to acquire preset live parameters, specify an object to be close-up at a live time specified by the live parameters as the object to be close-up, and the object to be close-up is any one of a host or a commodity; when the object to be close-up is the host, human body recognition is performed on the host to obtain the object frame and read coordinates of the object frame on the main video stream as the object frame coordinates; and when the object to be close-up is the commodity, the object frame is constructed according to an outline of the commodity, and coordinates of the object frame on the main video stream are read as the object frame coordinates.

[0067] Optionally, the intelligent identification module is configured to perform down-sampling processing on the main video stream to obtain a down-sampled main video stream; perform image recognition on the processed main video stream by using a human body detection model to determine a plurality of human body key points and an object frame of the close-up object in the main video stream.

[0068] Optionally, the intelligent identification module is configured to perform down-sampling processing on the main video stream to obtain a down-sampled main video stream; identify a commodity in the processed main video stream, and take the commodity as the close-up object, wherein the commodity is pasted with a commodity identification code or is consistent with a commodity picture uploaded by the host; identify an outline of the close-up object, and construct an object frame of the close-up object according to the outline.

[0069] Optionally, the adjustment module is configured to control the sub-camera to rotate in a direction of the close-up object based on the object frame coordinates, and adjust a shooting angle of the sub-camera; control the sub-camera to perform object identification point detection by using an object detection model while controlling the sub-camera to rotate, wherein the object identification point is a human head or a commodity center point; when the object identification point is detected based on the sub-camera, continuously acquire identification point coordinates of the object identification point in an image currently collected by the sub-camera; continuously calculate a position offset amount between the object frame and a camera visual center of the sub-camera based on the identification point coordinates; continue to control the sub-camera to rotate according to the position offset amount, until the position offset amount is less than an offset amount threshold, and stop rotating, so that the sub-camera faces the close-up object; control the sub-camera to collect an image of a currently faced direction to obtain the auxiliary video stream.

[0070] Optionally, the adjustment module is configured to read visual center coordinates of the camera visual center, calculate an Euclidean distance between the identification point coordinates and the visual center coordinates; construct a camera visual frame with the camera visual center as a center point, calculate an intersection-over-union between the camera visual frame and the object frame; query a first weight value corresponding to the Euclidean distance and a second weight value corresponding to the intersection-over-union, and perform weighted calculation on the Euclidean distance and the intersection-over-union by using the first weight value and the second weight value, and take a calculation result obtained as a position offset amount between the object frame and a camera visual center of the sub-camera.

[0071] Optionally, the director module is configured to detect whether a projectable region exists in an image frame of the main video stream; when it is detected that the projectable region exists in the image frame of the main video stream, a picture-in-picture region is formed by using the projectable region, the auxiliary video stream is projected in the picture-in-picture close-up region in the main video stream for playing, and the live picture is obtained; when it is detected that the projectable region does not exist in the image frame of the main video stream, a target sub-window is selected as the picture-in-picture region in the image frame of the main video stream, the auxiliary video stream is projected in the picture-in-picture close-up region in the main video stream for playing, and the live picture is obtained.

[0072] Optionally, the director module is configured to recognize a polygon region of a preset color in the image frame of the main video stream by using an image segmentation model, take the polygon region as the projectable region, generate an image mask with an area consistent with that of the projectable region, and superimpose the image mask on the projectable region to form a picture-in-picture close-up region; or, recognize an augmented reality (AR) identification code in the image frame of the main video stream, take a region covered by the AR identification code as the projectable region, generate an image mask with an area consistent with that of the projectable region, and superimpose the image mask on the projectable region to form a picture-in-picture close-up region.

[0073] Optionally, the director module is configured to read a host anchor coordinate of a host anchor and a product coordinate of a product in the main video stream, divide a to-be-displayed region including the host anchor and the product in the main video stream according to the host anchor coordinate and the product coordinate, set a specified number of sub-windows on the image frame of the main video stream, respectively calculate an intersection-over-union ratio between each sub-window and the to-be-displayed region, select a target sub-window with a minimum intersection-over-union ratio and less than an intersection-over-union ratio threshold from the specified number of sub-windows, and take the target sub-window as the picture-in-picture region.

[0074] Optionally, the director module is further configured to, when the to-be-close-up object is the host anchor, recognize a product with a product identification code pasted thereon in the main video stream, generate a combined feature vector, and input the combined feature vector into a preset pose classifier, where the combined feature vector is used to represent a positional relationship between a hand key point of the host anchor and the product; if an identification result output by the preset pose classifier indicates that the hand of the host anchor is in contact with the product, divide a to-be-displayed region including the host anchor and the product in the main video stream according to the host anchor coordinate and the product coordinate, and crop and scale the main video stream according to the to-be-displayed region to obtain the processed main video stream.

[0075] Optionally, the director module is configured to identify a plurality of human body key points of the host, and obtain a product identification code coordinate of the product identification code in the host video stream; a human body coordinate system is established with a top-left vertex of the object frame as an origin, the product identification code coordinate is converted, an image coordinate system in which the product identification code coordinate is located is transformed to the human body coordinate system, and the product identification code coordinate after the transformation is normalized to obtain a first normalized coordinate; a plurality of key point coordinates corresponding to the plurality of human body key points are converted, an image coordinate system in which the plurality of key point coordinates are located is transformed to the human body coordinate system, and the plurality of key point coordinates after the transformation are normalized respectively to obtain a plurality of second normalized coordinates; the first normalized coordinate and the plurality of second normalized coordinates are combined to generate a feature vector, and the feature vector is taken as a combined feature vector.

[0076] Optionally, the director module is further configured to output the live picture to a viewer end; when receiving a playing adjustment instruction of the viewer end, determine a to-be-played content indicated by the playing adjustment instruction; adjust the live picture according to the to-be-played content, and output the live picture after the adjustment to the viewer end.

[0077] Optionally, the director module is configured to, when the to-be-played content is a host camera image, take the host video stream as the live picture, and output the live picture to the viewer end, so that the viewer end plays an image picture of the host video stream in full screen; when the to-be-played content is a close-up image, take the auxiliary video stream as the live picture, and output the live picture to the viewer end, so that the viewer end plays an image picture of the auxiliary video stream in full screen; when the to-be-played content is a split-screen image, create a split-screen picture, divide a first projection area and a second projection area on the split-screen picture, project the host video stream to the first projection area, project the auxiliary video stream to the second projection area, and output the split-screen picture after the projection as the live picture, output the live picture to the viewer end, so that the viewer end plays the split-screen picture after the projection in full screen.

[0078] According to a third aspect of the present application, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in any of the first aspect when executing the computer program.

[0079] According to a fourth aspect of the present application, a readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the method in any of the first aspect when executed by a processor.

[0080] By the technical scheme, the application provides a live picture output method, device, computer equipment and readable storage medium. The application responds to a live start instruction, performs real-time image collection on a current live scene based on a main camera and a sub-camera, acquires a main video stream collected by the main camera, determines a to-be-close-up object, performs image recognition on the main video stream, determines object frame coordinates of the object frame of the to-be-close-up object on the main video stream, adjusts a shooting angle of the sub-camera according to the object frame coordinates, controls the sub-camera to face the to-be-close-up object and perform image collection on the to-be-close-up object, acquires a sub-video stream, determines a picture-in-picture close-up area in an image picture of the main video stream, projects an image picture of the sub-video stream to the picture-in-picture close-up area, acquires a live picture, and collects video images based on the double cameras. The collection of the double cameras can not only acquire a panoramic picture, but also perform close-up shooting on a selected object, so as to provide a live picture with different angles for a user. The live picture shows more diverse content and provides multi-angle live information, so that a user watching the live picture can understand more information about the live content and understand the live content in more detail.

[0081] The above description is only a summary of the technical scheme of the application. In order to make the technical means of the application more clear and understandable, the specific embodiments of the application are described below according to the content of the description, and in order to make the above and other purposes, features and advantages of the application more obvious and easy to understand. BRIEF DESCRIPTION OF DRAWINGS

[0082] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the scope of the present application. Furthermore, the same reference numerals are intended to identify the same components throughout the accompanying drawings. In the drawings:

[0083] Figure 1 A live picture output method flowchart provided by an embodiment of the application is shown;

[0084] Figure 2A Another live picture output method flowchart provided by an embodiment of the application is shown;

[0085] Figure 2B A live picture output method provided by an embodiment of the application is shown;

[0086] Figure 2C A live picture output method provided by an embodiment of the application is shown;

[0087] Figure 3A Another live picture output method flowchart provided by an embodiment of the application is shown;

[0088] Figure 3B A schematic diagram of a live picture output method provided by an embodiment of the present application is shown.

[0089] Figure 4 A structural schematic diagram of a live picture output device provided by an embodiment of the present application is shown.

[0090] Figure 5 A device structural schematic diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0091] Exemplary embodiments of the present application will be described in detail with reference to the drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thoroughly and completely understood, and so that the scope of the present application will be completely conveyed to those skilled in the art.

[0092] Applicants have realized that most existing ordinary live video devices are based on single cameras, and there are few models on the market that are based on dual cameras or multiple cameras and can effectively utilize the functions of multiple cameras in combination with AR (Augmented Reality) and AI (Artificial Intelligence) functions. When used, a certain professional knowledge is required for scene arrangement and installation, resulting in that the live picture that can be seen by the user at the audience end is usually a two-dimensional, single-angle picture, and the amount of information provided by the live picture to the user is limited. For some shopping live broadcasts, the user has difficulty in carefully understanding the goods through the live picture, and the different needs of many users cannot be met. Therefore, the present application proposes a live picture output method, which collects video images based on dual cameras, can not only collect panoramic pictures, but also can shoot close-ups of selected objects, provides live pictures of different angles for users, and makes the content displayed by the live picture more diverse and provides multi-directional live information, so that the user watching the live broadcast can understand more information about the live content through the live broadcast and understand the live content more carefully.

[0093] An embodiment of the present application provides a live picture output method, as shown in Figure 1 The method comprises the following steps.

[0094] 101、In response to a live start instruction, real-time image collection is performed on a current live scene based on a main camera and a sub-camera, and a main video stream collected by the main camera is acquired.

[0095] In the embodiment of the present application, the anchor can start live streaming based on the anchor end, which can be a terminal device such as a mobile phone or a tablet computer used for live streaming held by the anchor. The anchor end is equipped with a main camera and a sub-camera. The main camera is equipped with a conventional wide-angle lens, and the sub-camera is equipped with a long-focus lens. The sub-camera is equipped with a mechanical pan-tilt or a mirror turning mechanical brake device or other devices that can realize the tracking and shooting of the target by the sub-camera. An embedded processing chip with AI acceleration capability is equipped, and the software supports the Android (an operating system) system ecosystem. In the embodiment of the present application, the anchor end is equipped with one main camera and one sub-camera, and in actual application, the anchor end can also be equipped with multiple sub-cameras. The number of sub-cameras is not limited in the present application. The audience end is a terminal held by a user watching the live streaming content of the anchor, which can be a terminal device such as a mobile phone or a tablet computer held by the user.

[0096] The anchor can perform live streaming for users by downloading a client with live streaming function on the anchor end. The live streaming content can be talent show, outdoor exploration, online shopping, etc. In the embodiment of the present application, the live streaming content is taken as an example of online shopping. After successful downloading of the client, the anchor will start live streaming by operating the client to request starting live streaming. In this way, the anchor end will detect the live streaming start instruction initiated by the anchor. Further, since the present application uses dual cameras for live streaming to ensure that users on the audience end can see live streaming pictures from different angles, in response to the live streaming start instruction, the anchor end will perform real-time image collection on the current live streaming scene based on the main camera and the sub-camera, and obtain the main video stream collected by the main camera, so as to facilitate subsequent identification of the main video stream to determine how to adjust the sub-camera to realize close-up shooting of the object in the live streaming scene.

[0097] 102, determining the object to be close-up, and performing image recognition on the main video stream to determine the object frame coordinates of the object frame of the object to be close-up on the main video stream.

[0098] The object to be close-up is an object that needs to be enlarged and shot in the live streaming process to enable users on the audience end to see more details. It can be the anchor or the goods introduced by the anchor. The object to be close-up can be set by the anchor on the anchor end. Therefore, when the anchor requests live streaming, the anchor end will determine the object to be close-up, perform image recognition on the main video stream, and determine the object frame coordinates of the object frame of the object to be close-up on the main video stream, so as to determine the position of the object to be close-up and perform close-up shooting on the object to be close-up. The object frame is a frame used to indicate the position of the object to be close-up, which can be a dashed frame, a solid frame or other quadrilateral frame. The object to be close-up can be framed in the main video stream by the object frame, so as to adjust the shooting angle of the sub-camera according to the object frame.

[0099] 103. Adjust a shooting angle of the secondary camera according to the object frame coordinates, control the secondary camera to face the object to be close-up and collect images of the object to be close-up, and obtain a secondary video stream.

[0100] After the object frame coordinates of the object frame of the object to be close-up on the main video stream are determined, the anchor end adjusts the shooting angle of the secondary camera according to the object frame coordinates, controls the secondary camera to face the object to be close-up and collect images of the object to be close-up, and obtains a secondary video stream. In this way, not only the panoramic shooting of the live scene is realized through the primary camera, but also the close-up shooting of the object in the live scene is realized through the secondary camera, so that the user at the audience end can understand the live content in more detail through the close-up shooting.

[0101] 104. Determine a picture-in-picture close-up area in the image frame of the main video stream, project the image frame of the secondary video stream to the picture-in-picture close-up area, and obtain a live picture.

[0102] After the main video stream obtained by the panoramic collection of the live scene by the primary camera and the secondary video stream obtained by the close-up collection of the object to be close-up by the secondary camera are obtained, the anchor end determines a picture-in-picture close-up area in the image frame of the main video stream, projects the image frame of the secondary video stream to the picture-in-picture close-up area, and obtains a live picture. Specifically, a small window can be set on the image frame of the main video stream, the small window is used as the picture-in-picture close-up area, the image frame of the secondary video stream is projected to the small window, the image frame of the secondary video stream is played on the small window, and the synthesis of the main video stream and the secondary video stream is realized; or a close-up button can be set on the image frame of the main video stream as the live picture, when it is detected that the user triggers the close-up button, a small window is created in the image frame of the main video as the picture-in-picture close-up area and the secondary video stream is put in, the manual picture-in-picture operation of the user is realized, the normal viewing of the user is avoided from being affected by the picture-in-picture close-up area, and the user can understand the details of the object; or it can be detected in real time whether the object to be close-up is triggered at the audience end, and when it is detected that the object to be close-up is triggered, a small window is created in the image frame of the main video as the picture-in-picture close-up area and the secondary video stream is put in. The timing and manner of determining the picture-in-picture close-up area are not limited in the application.

[0103] The method provided in the embodiments of the present application, in response to a live broadcast starting instruction, performs real-time image acquisition on a current live broadcast scene based on a main camera and a secondary camera, acquires a main video stream collected by the main camera, determines a to-be-close-up object, and performs image recognition on the main video stream to determine object frame coordinates of an object frame of the to-be-close-up object on the main video stream, adjusts a shooting angle of the secondary camera according to the object frame coordinates, controls the secondary camera to face the to-be-close-up object and perform image acquisition on the to-be-close-up object to obtain a secondary video stream, determines a picture-in-picture close-up area in an image frame of the main video stream, projects an image frame of the secondary video stream to the picture-in-picture close-up area, and obtains a live broadcast picture. The acquisition of the video image based on the dual cameras can not only acquire a panoramic picture, but also perform close-up shooting on a selected object, thereby providing the user with live broadcast pictures of different angles, making the content displayed in the live broadcast picture more diverse, and providing multi-angle live broadcast information, so that the user watching the live broadcast can learn more information about the live broadcast content and understand the live broadcast content in more detail.

[0104] Further, as a refinement and expansion of the foregoing embodiment, in order to completely describe the specific implementation process of the embodiment, the embodiment of the present application provides another method for outputting a live broadcast picture, as shown in Figure 2A The method comprises the following steps.

[0105] 201. In response to a live broadcast starting instruction, real-time image acquisition is performed on a current live broadcast scene based on a main camera and a secondary camera, and a main video stream collected by the main camera is acquired.

[0106] The embodiment of the present application relates to the interaction between the anchor end and the audience end. The anchor end can be a terminal device for live broadcast such as a mobile phone or a tablet computer held by an anchor. The anchor end is equipped with a main camera and a secondary camera. The main camera is equipped with a conventional wide-angle lens, the secondary camera is equipped with a long-focus lens, and the secondary camera is equipped with a mechanical pan-tilt or a mirror turning mechanical brake device or other devices that can realize target tracking and shooting of the secondary camera. An embedded processing chip with AI acceleration capability is equipped, and the software supports the Android system ecology. In the embodiment of the present application, the anchor end is equipped with one main camera and one secondary camera as an example, and in the actual application process, the anchor end can also be equipped with multiple secondary cameras, and the number of secondary cameras is not limited in the present application. The audience end is a terminal held by a user watching the live broadcast content of the anchor, and can be a terminal device such as a mobile phone or a tablet computer held by the user.

[0107] After the client is successfully downloaded, the host will request to start the live broadcast by operating on the client, so that the host end will detect the live broadcast start instruction initiated by the host. In response to the live broadcast start instruction, the host end will perform real-time image acquisition on the current live broadcast scene based on the main camera and the auxiliary camera, and obtain the main video stream collected by the main camera, so as to facilitate subsequent identification of the main video stream to determine how to adjust the auxiliary camera to realize close-up of the object in the live broadcast scene. It should be noted that in the actual application process, in order to ensure that the videos collected by the main camera and the auxiliary camera are synchronized, the auxiliary video stream collected by the auxiliary camera can also be synchronized to obtain and cache.

[0108] 202, determine the object to be close-up, and perform image recognition on the main video stream to determine the object frame coordinates of the object frame of the object to be close-up on the main video stream.

[0109] In the embodiment of the present application, the object to be close-up is an object that needs to be enlarged for shooting in the live broadcast process so that the user at the audience end can see more details. Specifically, it can be the host or the goods introduced by the host. The object to be close-up can be set by the host on the host end. Therefore, when the host requests to live broadcast, the host end will determine the object to be close-up, perform image recognition on the main video stream, and determine the object frame coordinates of the object frame of the object to be close-up on the main video stream, so as to determine the position of the object to be close-up, and thus perform close-up shooting on the object to be close-up. The object frame is a frame used to indicate the position of the object to be close-up, which can be a dashed frame, a solid frame, or other quadrilateral frames. Specifically, the object to be close-up can be framed in the main video stream by using the object frame, so as to adjust the shooting angle of the auxiliary camera according to the object frame.

[0110] Since the object to be close-up can be the host or the goods in the embodiment of the present application, the process of determining the object frame coordinates for different objects to be close-up is described as follows:

[0111] Before the live broadcast starts, the host will set which object as the close-up object during the live broadcast in advance on the host end, and the setting will be saved in the live broadcast parameters of the host end. That is, the preset live broadcast parameters will include the object to be close-up, which is any one of the host or the goods. Therefore, the host end will first obtain the preset live broadcast parameters, and take the object specified by the live broadcast parameters as the object to be close-up during the live broadcast.

[0112] Further, when the to-be-close-up object is the host, the host end performs human body recognition on the host to obtain an object frame and reads coordinates of the object frame on the main video stream as object frame coordinates. Specifically, before performing the human body recognition, the host end performs downsampling processing on the obtained main video stream to obtain a main video stream after downsampling processing. The reason for performing the downsampling processing is that AI detection of a live streaming scene does not have a particularly high requirement on resolution, and the AI detection will be performed on a small-resolution image, which is helpful to reduce the algorithm consumption of the embedded end. The host end can use a human body detection model to perform image recognition on the main video stream to determine multiple human body key points and the object frame of the to-be-close-up object in the main video stream. The human body detection model can be a deep learning detection model constructed using a YOLOv5 (a kind of neural network model) architecture, which is used to detect human bodies and key points on human bodies; YOLOv5 is a deep learning-based target detection algorithm, which uses a lightweight network architecture mainly composed of convolutional layers, pooling layers, connection layers and final detection layers, and adopts CSPDarknet53 (a kind of network structure) as a basic backbone network and adds a feature aggregation module and an effective downsampling strategy to extract rich feature representations. When training the human body detection model, first, image datasets containing target objects need to be collected and labeled, and the labeling mainly includes the positions of object bounding boxes and classification labels, and the datasets are divided into a training set, a validation set and a test set. Subsequently, the datasets are used for iterative training of the model, the input image is forward propagated through the network to obtain a feature map, the feature map is matched with the real label, a loss function such as a cross-entropy loss and a mean square error loss commonly used in target detection tasks is calculated, and the network parameters are updated using a back propagation algorithm to constantly optimize the model, and finally the human body detection model is obtained. Therefore, using the human body detection model can realize recognition of human body key points and directly recognize the bounding box of the object, which is the object frame of the to-be-close-up object. In addition, in the actual application process, 17 key points on the human body can be predicted based on the human body detection model, and the 17 key points predicted are taken as the multiple human body key points.

[0113] When the object to be close-up is a commodity, the anchor end constructs an object box according to the contour of the commodity, and reads the coordinates of the object box on the main video stream as the object box coordinates. Specifically, before the commodity is identified, the anchor end can also perform down-sampling processing on the obtained main video stream to reduce the algorithm consumption of the embedded end. Then, the anchor end identifies the commodity in the processed main video stream, and takes the commodity as the object to be close-up. The commodity is pasted with a commodity identification code or consistent with the commodity picture uploaded by the anchor, that is, the anchor can paste a commodity identification code such as a brand logo sticker, a red logo sticker, a green logo sticker, Marker ID (Marker Industrial Design), etc. on the commodity in advance, so that the anchor end can quickly determine the commodity by identifying these commodity identification codes, or the anchor can upload a real shot picture of the commodity, so that the anchor end can determine which object is the commodity of the current live broadcast session by comparing the objects in the live broadcast scene. For example, the anchor can paste Aruco (a kind of binary square marker) on the commodity to be displayed, so that the anchor end can directly determine which object is the commodity of the current live broadcast by identifying Aruco, facilitating close-up of the commodity. After the commodity is determined and taken as the object to be close-up, the anchor end identifies the contour of the object to be close-up, and constructs an object box of the object to be close-up according to the contour.

[0114] In an optional embodiment, an intelligent recognition module can be arranged in the anchor end, in which a human body detection model and a commodity identification model are built-in, so that the intelligent recognition module has an AI recognition function, can detect people and commodities in the live broadcast scene, perform saliency analysis on the live broadcast scene, and recognize the body interaction action of the people to make corresponding response. In this way, the main camera and the auxiliary camera simultaneously collect image data of the live broadcast scene during live broadcast. In addition to the image data needing to be compressed and encoded in real time every frame and transmitted to the video distribution server through the streaming media protocol, the same content video data (including the main camera and the auxiliary camera, a total of 2 paths) are also needed to be down-sampled and sent to the intelligent recognition module in the system, and the intelligent recognition module performs the image recognition process in step 202.

[0115] 203、Taking the object box coordinates as the reference, the auxiliary camera is controlled to rotate towards the object to be close-up, and the auxiliary camera is controlled to perform object landmark detection while the auxiliary camera is rotating.

[0116] In the embodiment of the present application, after the object box coordinates are determined, the anchor end will take the object box coordinates as the reference to control the sub-camera to rotate towards the direction of the object to be close-up, and at the same time of controlling the sub-camera to rotate, control the sub-camera to detect the object identification point by using the object detection model. Among them, the object identification point is the head point of the human body or the center point of the commodity. Similarly, if the object identification point is the head point of the human body, a deep learning detection model based on YOLOv5 architecture can also be used to detect the object identification point. It should be noted that since only the position of the object identification point needs to be roughly determined here, the depth and parameter amount of the object detection model used here can also be simplified to some extent, thereby reducing the training pressure of the model while also improving the detection speed of the object identification point.

[0117] 204、When the object identification point is detected based on the sub-camera, the identification point coordinates of the object identification point in the image currently collected by the sub-camera are continuously acquired, and the position offset between the object box and the camera visual center of the sub-camera is continuously calculated based on the identification point coordinates, and the sub-camera is controlled to rotate, until the position offset is less than the offset threshold, and the sub-camera is controlled to stop rotating, so that the sub-camera faces the object to be close-up.

[0118] In the embodiment of the present application, when the anchor end detects the object identification point based on the sub-camera, the anchor end will continuously acquire the identification point coordinates of the object identification point in the image currently collected by the sub-camera, and continuously calculate the position offset between the object box and the camera visual center of the sub-camera based on the identification point coordinates, and control the sub-camera to rotate, until the position offset is less than the offset threshold, and control the sub-camera to stop rotating, so that the sub-camera faces the object to be close-up. That is, after detecting the object identification point, the sub-camera will continuously calculate the position offset, and gradually adjust the gimbal according to the position offset, until the position offset between the center of the object box and the camera visual center is less than a certain threshold, considering that the sub-camera has faced the object to be close-up, and stop rotating.

[0119] In an optional embodiment, the position offset is obtained by fusing the Euclidean distance between the object box and the camera visual center of the secondary camera and the intersection over union. The reason for using this fusion mechanism is that the applicant realizes that although the Euclidean distance can accurately measure the degree of offset of the object box from the camera visual center, it is too accurate and can easily cause jitter of the object box in actual scenarios. For example, when a human body box is large, that is, it slightly shakes left and right in the near distance, the corresponding Euclidean distance in the image coordinates will also change, causing instability in tracking. The intersection over union, that is, IoU (Intersection over Union), has a lower degree of accuracy than the Euclidean distance, but can improve the instability of the large human body detection target box in the near distance. Therefore, the embodiment of the present application proposes a fusion mechanism that combines the advantages and disadvantages of the two measurement methods to calculate the position offset. The calculation process of the position offset is described below:

[0120] First, the anchor end reads the visual center coordinates of the camera visual center, calculates the Euclidean distance between the identification point coordinates and the visual center coordinates. Assuming that the identification point coordinates are (x1, y1) and the visual center coordinates are (x2, y2), the Euclidean distance d between the identification point coordinates and the visual center coordinates can be calculated by the following formula 1:

[0121] Formula 1:

[0122] Subsequently, the anchor end constructs a camera visual frame with the camera visual center as the center point, and calculates the intersection over union between the camera visual frame and the object box. The intersection over union, that is, IoU, is a concept used in target detection. The IoU calculates the intersection rate of the "predicted bounding box" and the "real bounding box", that is, the ratio of their intersection and union. The most ideal case is complete overlap, that is, the ratio is 1. In the embodiment of the present application, the "predicted bounding box" is the camera visual frame, and the "real bounding box" is the object box. Assuming that the object box is represented by B1 and the camera visual frame is represented by B2, the intersection over union IoU between the camera visual frame and the object box can be calculated by the following formula 2:

[0123] Formula 2:

[0124] Finally, the anchor end queries the first weight value corresponding to the Euclidean distance and the second weight value corresponding to the IoU, and performs weighted calculation on the Euclidean distance and the IoU by using the first weight value and the second weight value, and takes the obtained calculation result as the position offset between the object frame and the camera visual center of the secondary camera. That is, the Euclidean distance and the IoU are weighted calculated according to the weight values corresponding to the two, and the calculated result is taken as the position offset. In the actual application, the weight values corresponding to the Euclidean distance and the IoU can be set by the developer, and for some small objects to be close-up, the weight of the Euclidean distance can be set to be larger, and the weight of the IoU can be set to be smaller, and vice versa for larger objects to be close-up, so as to ensure the accuracy of the calculated position offset.

[0125] 205, control the secondary camera to collect images in the current facing direction, and obtain a secondary video stream.

[0126] In the embodiment of the present application, the above steps 203 and 204 are performed to realize the adjustment of the secondary camera according to the object frame coordinates, at this time, the secondary camera has been aligned to the object to be close-up, and the position of the object to be close-up is also known, at this time, the main video stream outputs the panoramic picture of the main camera, the secondary camera is controlled to face the object to be close-up and collect images of the object to be close-up, and a secondary video stream is obtained, and the secondary video stream outputs the panoramic picture of the secondary camera.

[0127] 206, determine a picture-in-picture close-up area in the image picture of the main video stream, project the image picture of the secondary video stream to the picture-in-picture close-up area, and obtain a live picture.

[0128] In order to enable different angle pictures to be reflected in the current played image picture in a novel form, many video playing platforms provide a picture-in-picture close-up mode, that is, a multi-angle close-up picture is projected on a certain area in the current played image picture, so that the two are played synchronously. Among them, the traditional picture-in-picture close-up mode is to set a small window to play a close-up picture at a fixed position such as the upper right corner, the upper left corner or the lower left corner of the current played picture, but the applicant realizes that this picture-in-picture close-up mode is relatively limited, if the anchor moves or someone in the live scene shows some objects, information, etc. at the edge of the picture, it will be blocked by the picture-in-picture area. Therefore, the present application proposes an intelligent picture-in-picture close-up mode, the anchor can place a television screen, a computer screen, a tablet computer and other preset color polygon areas in the live scene or set some AR markers, such as a cuboid with an AR checkerboard identification pasted thereon, as a projection area, so that the close-up picture is played on such area to avoid blocking the objects and personnel in the live scene; if the anchor does not place these projection objects in the live scene, the present application will also intelligently identify the anchor and the goods he is introducing, track the anchor and the goods, and create a close-up small window near the anchor and the goods, so that the small window playing the close-up video can move with the anchor and the goods, and will not block a certain fixed position for a long time, and has good flexibility. The process of the picture-in-picture close-up proposed by the present application will be described as follows:

[0129] Since whether the anchor places a polygon region and a marker in the live scene is the key to determining how to perform the picture-in-picture close-up projection, the anchor end first detects whether a projectable region exists in the image frame of the main video stream. The projectable region is the preset color polygon region of a television screen, a computer screen, a tablet computer, or some AR markers set by pasting AR marker codes, such as a cuboid with pasted AR checkerboard markers. On the one hand, when it is detected that the projectable region exists in the image frame of the main video stream, it indicates that the projectable region can be used for close-up projection, and there is no need to create a small window additionally. Therefore, the anchor end forms a picture-in-picture region by using the projectable region, projects the auxiliary video stream in the picture-in-picture close-up region in the main video stream, and obtains a live picture. Specifically, if it is detected that the projectable region is a preset color polygon region of a television screen, a computer screen, a tablet computer, or the like, the anchor end uses an image segmentation model to identify the polygon region of the preset color in the image frame of the main video stream, takes the polygon region as the projectable region, generates an image mask with an area consistent with that of the projectable region, and superimposes the image mask on the projectable region to form a picture-in-picture close-up region. The image mask can be a generated mask, that is, the mask is synthesized into a quadrilateral picture-in-picture close-up region. If it is detected that an object with pasted AR marker codes exists, the anchor end can identify the AR marker codes in the image frame of the main video stream, take the region covered by the AR marker codes as the projectable region, generate an image mask with an area consistent with that of the projectable region, and superimpose the image mask on the projectable region to form a picture-in-picture close-up region. For example, assuming that the anchor end detects a paper box with pasted AR checkerboard markers in the image frame of the main video stream, the anchor end can count the region covered by the AR checkerboard markers on the paper box and take the region as the projectable region.

[0130] In another aspect, when it is detected that there is no projectable region in the image frame of the main video stream, it indicates that there is no object that can be projected in the current live scene, and the anchor end needs to create a small window following the anchor and the commodity to play a close-up. Therefore, the anchor end selects a target sub-window as the picture-in-picture region in the image frame of the main video stream, projects the auxiliary video stream in the picture-in-picture close-up region in the main video stream to obtain a live picture. Specifically, when selecting the target sub-window, in order to enable the target sub-window to follow the anchor and the commodity at the same time, the anchor end reads the anchor coordinates of the anchor and the commodity coordinates of the commodity in the main video stream, and divides a to-be-displayed region including the anchor and the commodity in the main video stream according to the anchor coordinates and the commodity coordinates. Subsequently, the anchor end sets a specified number of sub-windows on the image frame of the main video stream, calculates the intersection-over-union between each sub-window and the to-be-displayed region respectively, and selects a target sub-window with the smallest intersection-over-union with the to-be-displayed region and less than an intersection-over-union threshold from the specified number of sub-windows, and takes the target sub-window as the picture-in-picture region. In this way, continuously calculating the intersection-over-union and selecting the target sub-window enables the selected target sub-window to move along with the to-be-displayed region where the anchor and the commodity are located, avoiding that the target sub-window blocks a certain fixed position for a long time, and good flexibility is achieved. In addition, the process of calculating the intersection-over-union between the sub-window and the to-be-displayed region is consistent with the process of calculating the intersection-over-union described in step 204 above, and will not be described again here.

[0131] 207、outputting the live picture to the audience end, when receiving a play adjustment instruction of the audience end, determining to-be-played content indicated by the play adjustment instruction, adjusting the live picture according to the to-be-played content, and outputting the adjusted live picture to the audience end.

[0132] In the embodiment of the present application, after obtaining the live picture, the anchor end outputs the live picture to the audience end to enable the audience end to play the live picture.

[0133] It should be noted that, since the picture-in-picture close-up region is used to play the close-up picture, and the main camera and the auxiliary camera both continuously collect videos in real time, in the embodiment of the present application, for a user of the audience end, the user can request the large screen to play the close-up picture by clicking the picture-in-picture close-up region, or can request to play only the picture collected by the main camera without playing the close-up picture, or can request to play the pictures of the main video stream and the auxiliary video stream in split screen.

[0134] Specifically, the user at the audience end can send a play adjustment instruction to the anchor end by triggering a corresponding button in the live application, so that the anchor end outputs a video stream to the audience end according to the to-be-played content indicated by the play adjustment instruction. When the to-be-played content is the main camera image, it indicates that the user at the audience end wants to see only the content captured by the main camera and does not want to see the close-up content captured by the auxiliary camera. Therefore, the anchor end takes the main video stream as the live picture and outputs the live picture to the audience end, so that the audience end plays the image picture of the main video stream in full screen. When the to-be-played content is the close-up image, it indicates that the user at the audience end currently wants to see only the close-up picture captured by the auxiliary camera and does not want to see the panoramic image of the live room captured by the main camera. Therefore, the anchor end takes the auxiliary video stream as the live picture and outputs the live picture to the audience end, so that the audience end plays the image picture of the auxiliary video stream in full screen. When the to-be-played content is the split-screen image, it indicates that the user at the audience end wants to see the main and auxiliary video streams at the same time and wants to view them in the form of split screen. Therefore, the anchor end creates a split-screen picture, divides a first projection area and a second projection area on the split-screen picture, projects the main video stream to the first projection area, projects the auxiliary video stream to the second projection area, takes the projected split-screen picture as the live picture, and outputs the live picture to the audience end, so that the audience end plays the projected split-screen picture in full screen.

[0135] Through the above process, the live picture can be switched according to the needs of the user, which facilitates the user to understand the panorama of the live room and the close-up of the anchor and the goods, and meets the various needs of different users. In actual application, a director module can be arranged in the anchor end, and the director module has an AR function. The director module performs the operations in steps 206 and 207 to generate the live picture and output the live picture.

[0136] In summary, the process of the output method of the live picture proposed in the embodiments of the present application is as follows:

[0137] For example, the anchor end is provided with an intelligent recognition module and a director module, as shown in Figure 2BAs shown, the anchor end is equipped with a main camera and a sub-camera, and the sub-camera can be controlled based on a holder. The anchor end starts the main camera and the sub-camera to collect live content images, obtains a main video stream and an auxiliary video stream, and performs intelligent AI recognition on the main video stream to realize rotation of the sub-camera, so that the sub-camera is aimed at the anchor for shooting. At the same time, the two video streams are sent to an intelligent recognition module and a director module. The intelligent recognition module realizes detection of the anchor, gestures, AR identification codes, goods, and picture-in-picture regions through recognition of the two video streams, so that the director module realizes anchor close-up synthesis, goods close-up synthesis, picture-in-picture display, and holder automatic tracking according to the recognized content, thereby generating a live picture and sending the live picture to a transmission layer. The transmission layer has a video distribution function and can specifically include a streaming server, a signaling server, etc. The video is transmitted by these servers based on RTMP (Time Messaging Protocol) and RTSP (Real Time Streaming Protocol) protocols, transmitted to the audience end, and video distribution is realized. After receiving the live picture, the audience end can realize playing of the live picture based on double-camera video decoding and AR picture-in-picture frame anchoring, and realize panoramic preview, anchor close-up, goods close-up, and split-screen display of the live picture when a user's triggering operation is detected.

[0138] Further, taking the sub-camera close-up anchor as an example, the live picture seen by the audience end can specifically refer to the example picture in Figure 2C . First, as shown in Figure 2C ①, the audience end is playing the main video stream collected by the main camera. In the picture of the main video stream, the anchor is introducing the goods to be demonstrated on the desktop, and a goods identification code is pasted on the goods to be demonstrated, which is used to indicate the position of the goods to be demonstrated. An electronic screen for picture-in-picture projection is placed behind the anchor, and a goods close-up video synthesized by the anchor end in the main video stream is played on the electronic screen. Subsequently, when the user of the audience end triggers the anchor to request a close-up of the anchor, the audience end will switch to full-screen playing of the anchor close-up picture in the auxiliary video stream, as shown in Figure 2C ②; and when the user of the audience end clicks the electronic screen for picture-in-picture projection, the audience end will switch to panoramic playing of the image data projected in the picture-in-picture on the electronic screen, as shown in Figure 2C ③, so that the user of the audience end can understand the detailed information of the goods.

[0139] The method provided by the embodiment of the application can collect panoramic pictures based on the dual cameras, can perform close-up shooting on a selected object, and can provide live pictures of different angles for the user, so that the content displayed by the live pictures is more diverse, and multi-angle live information is provided, so that the user watching the live broadcast can learn more information about the live content and understand the live content in more detail.

[0140] Further, considering that the host will introduce the goods in detail by touching the goods during the live broadcast, in order to enable the host end to automatically perform close-up shooting on the goods when the host introduces the goods in detail, the embodiment of the application provides another method for outputting a live picture, and realizes intelligent tracking framing. The gimbal is called for automatic tracking by detecting a hand or a product identification code attached to the product, as shown in Figure 3A The method comprises the following steps.

[0141] 301. In response to a live broadcast start instruction, real-time image collection is performed on a current live broadcast scene based on a main camera and a sub-camera, and a main video stream collected by the main camera is acquired.

[0142] The specific execution process and the detailed content are the same as those described in the above step 201, and will not be described here again.

[0143] 302. A to-be-close-up object is determined, and image recognition is performed on the main video stream to determine object frame coordinates of the object frame of the to-be-close-up object on the main video stream.

[0144] The specific execution process and the detailed content are the same as those described in the above step 202, and will not be described here again.

[0145] 303. The sub-camera is controlled to rotate towards the to-be-close-up object based on the object frame coordinates, and the sub-camera is controlled to perform object identification point detection based on the object detection model while the sub-camera is rotating.

[0146] The specific execution process and the detailed content are the same as those described in the above step 203, and will not be described here again.

[0147] 304. When the object identification point is detected based on the sub-camera, identification point coordinates of the object identification point in an image currently collected by the sub-camera are continuously acquired, and the position offset between the object frame and the camera visual center of the sub-camera is continuously calculated based on the identification point coordinates, and the sub-camera is controlled to rotate until the position offset is less than an offset threshold, and the sub-camera is controlled to stop rotating, so that the sub-camera faces the to-be-close-up object.

[0148] The specific execution process and the detailed content are the same as those described in the above step 204, and will not be described here again.

[0149] 305. The auxiliary camera controls image acquisition of the current facing direction to obtain an auxiliary video stream.

[0150] The specific implementation process and details are consistent with the description in step 205, and will not be repeated here.

[0151] 306. A picture-in-picture close-up area is determined in the image frame of the main video stream, and the image frame of the auxiliary video stream is projected into the picture-in-picture close-up area to obtain a live picture.

[0152] The specific implementation process and details are consistent with the description in step 206, and will not be repeated here.

[0153] 307. When the object to be close-up is the host, the host end identifies the product with the product identification code in the main video stream, generates a combined feature vector representing the positional relationship between the hand key points of the host and the product, and inputs the combined feature vector into the preset posture classifier.

[0154] In the embodiments of the present application, since the contact between the hand of the host and the product can be identified only when the host is close-up, in the embodiments of the present application, when the object to be close-up is the host, the host end identifies the product with the product identification code in the main video stream, generates a combined feature vector representing the positional relationship between the hand key points of the host and the product, inputs the combined feature vector into the preset posture classifier, and identifies the combined feature vector by using the preset posture classifier to determine whether the hand of the host is in contact with the product and further determine whether the host is introducing the product.

[0155] The generation manner of the combined feature vector is described as follows: first, the anchor end identifies multiple human body key points of the anchor, and obtains the product identification code coordinates of the product identification code in the main video stream. The process of identifying multiple human body key points with the product identification code coordinates is consistent with the process described in step 202, and will not be described here. Subsequently, the anchor end establishes a human body coordinate system with the top-left vertex of the object frame as the origin, converts the coordinates of the product identification code, transforms the image coordinate system where the product identification code coordinates are located to the human body coordinate system, and normalizes the converted product identification code coordinates to obtain first normalized coordinates. Next, the anchor end converts the coordinates of the multiple key points corresponding to the multiple human body key points, transforms the image coordinate system where the multiple key points are located to the human body coordinate system, and normalizes the converted multiple key points coordinates respectively to obtain multiple second normalized coordinates. It should be noted that the multiple human body key points can be the key points of the human hand and shoulder, and the number of selected key points depends on the human body detection model, but the more key points, the better the effect. Finally, the anchor end combines the first normalized coordinates and the multiple second normalized coordinates to generate a feature vector and takes the feature vector as a combined feature vector.

[0156] For example, assuming that the position of the human body detection frame is (x1, y1, x2, y2), x1, y1, ~ x k , and y k are the normalized coordinates of the key points of the human hand and shoulder in the human body coordinate system, and mx1, my1 are the normalized coordinates of the product in the human body coordinate system, then the generated combined feature vector v is v = (x1, y1, …, x k , y k , mx1, my1).

[0157] After generating the combined feature vector, the anchor end inputs the combined feature vector into the preset posture classifier, identifies the combined feature vector using the preset posture classifier, and determines whether the hand of the anchor and the product are in contact, and further determines whether the anchor is introducing the product.

[0158] 308、If the identification result output by the preset posture classifier indicates that the hand of the anchor and the product are in contact, a to-be-displayed region including the anchor and the product is divided in the main video stream according to the anchor coordinates and the product coordinates, and the main video stream is cropped and scaled according to the to-be-displayed region to obtain a processed main video stream.

[0159] In the embodiments of the present application, the preset posture classifier can be an SVM posture classifier, which can be obtained by using the same data preprocessing method as the generation method of the human body detection model described in step 202, which will not be repeated here. Further, after inputting the combined feature vector into the preset posture classifier, the preset posture classifier will output two recognition results, 0 and 1 respectively; wherein 0 indicates that the current anchor is not showing the goods, and 1 indicates that the current anchor's hand is in contact with the goods, indicating that it is showing the goods.

[0160] When the recognition result indicates that the anchor's hand is in contact with the goods, that is, the recognition result is 1, the anchor end will obtain the anchor coordinates of the anchor in the main video stream and the goods coordinates of the goods in the main video stream, and divide a to-be-displayed region including the anchor and the goods in the main video stream according to the anchor coordinates and the goods coordinates, that is, the anchor end will perform saliency analysis on the panoramic image of the main camera combined with the goods identification code coordinates of the goods identification code of the goods, and synthesize a close-up region of the anchor plus the goods, so that the anchor and the goods can both be played in close-up. Subsequently, the anchor end crops and scales the main video stream according to the to-be-displayed region to obtain a processed main video stream. In this way, through the above process, the close-up and enlargement of the main video stream are realized. The anchor end will output the processed main video stream to the audience end in the subsequent process, so that the audience end plays the processed main video stream. In this way, during the live broadcast, if the anchor starts to introduce the goods, the user of the audience end will automatically see the details of the goods introduced by the anchor, and even can see the detailed action of the anchor's hand when introducing the goods. Moreover, the switching of these pictures is automatically performed by the anchor end. Compared with the traditional cumbersome method of manually adjusting the camera angle, the anchor does not need to manually request a close-up of the goods, and the user of the audience end does not need to manually touch the screen to switch, which automatically extracts the content concerned by the user of the audience end for presentation, and has good intelligence.

[0161] Further, taking the close-up of the anchor by the secondary camera as an example, the live broadcast picture seen by the audience end can refer to the example picture in Figure 3B Figure 3B Firstly, as shown in FIG. 1 of Figure 3B ​The auxiliary video stream captured by the secondary camera is played in full screen, a target sub-window is selected in the image picture of the full-screen played auxiliary video stream, and the reduced main video stream is played on the target sub-window, so that the close-up content of the auxiliary video stream and the zoomed main video stream can be presented to the user at the audience end at the same time. Thus, when the anchor introduces the goods, automatic lens switching and close-up display are realized, the user at the audience end can learn the details of the goods in time, the user can avoid missing the display of details, and the intelligence is good. The application does not specifically limit whether the auxiliary video stream and the zoomed main video stream are presented at the same time.

[0162] The method provided by the embodiment of the application can not only collect panoramic pictures but also perform close-up shooting on selected objects, so as to provide live pictures of different angles for the user, the content displayed by the live pictures is more diverse, and the live information is provided from multiple directions, so that the user watching the live broadcast can learn more information about the live content and understand the live content in more detail.

[0163] Further, as Figure 1 The embodiment of the application provides an output device of a live picture, as shown in the method. Figure 4 The device includes an acquisition module 401, an intelligent identification module 402, an adjustment module 403, and a director module 404.

[0164] The acquisition module 401 is configured to, in response to a live start instruction, perform real-time image acquisition on a current live scene based on a primary camera and a secondary camera, and acquire a main video stream collected by the primary camera.

[0165] The intelligent identification module 402 is configured to determine a to-be-close-up object, and perform image identification on the main video stream to determine object frame coordinates of the object frame of the to-be-close-up object on the main video stream.

[0166] The adjustment module 403 is configured to adjust a shooting angle of the secondary camera according to the object frame coordinates, control the secondary camera to face the to-be-close-up object and perform image acquisition on the to-be-close-up object, and obtain an auxiliary video stream.

[0167] The director module 404 is configured to determine a picture-in-picture close-up area in an image picture of the main video stream, project an image picture of the auxiliary video stream to the picture-in-picture close-up area, and obtain a live picture.

[0168] In a specific application scenario, the intelligent identification module 402 is configured to acquire preset live broadcast parameters, and specify an object in a close-up shot of the live broadcast at a specified time as the object to be shot, the object to be shot being any one of a host or a commodity; when the object to be shot is the host, the host is subjected to human body recognition to obtain an object frame and read coordinates of the object frame on the main video stream as object frame coordinates; when the object to be shot is the commodity, the object frame is constructed according to an outline of the commodity, and coordinates of the object frame on the main video stream are read as the object frame coordinates.

[0169] In a specific application scenario, the intelligent identification module 402 is configured to perform down-sampling processing on the main video stream to obtain the main video stream after down-sampling processing, and perform image recognition on the processed main video stream by using a human body detection model to determine multiple human body key points and an object frame of the object to be shot in the main video stream.

[0170] In a specific application scenario, the intelligent identification module 402 is configured to perform down-sampling processing on the main video stream to obtain the main video stream after down-sampling processing, and identify a commodity in the processed main video stream, the commodity being the object to be shot, the commodity having a commodity identification code affixed thereto or being consistent with a commodity picture uploaded by the host; an outline of the object to be shot is identified, and an object frame of the object to be shot is constructed according to the outline.

[0171] In a specific application scenario, the adjustment module 403 is configured to control the sub-camera to rotate in a direction of the object to be shot based on the object frame coordinates, and adjust a shooting angle of the sub-camera; while controlling the sub-camera to rotate, the sub-camera is controlled to perform object identification point detection by using an object detection model, the object identification point being a human head or a commodity center point; when the object identification point is detected based on the sub-camera, identification point coordinates of the object identification point in an image currently captured by the sub-camera are continuously acquired; based on the identification point coordinates, a positional offset between the object frame and a camera visual center of the sub-camera is continuously calculated; the sub-camera is continuously controlled to rotate according to the positional offset until the positional offset is less than an offset threshold, and the rotation is stopped, so that the sub-camera faces the object to be shot; the sub-camera is controlled to capture an image of a currently faced direction to obtain the auxiliary video stream.

[0172] In a specific application scenario, the adjustment module 403 is used to read the visual center coordinates of the camera's visual center, calculate the Euclidean distance between the marker point coordinates and the visual center coordinates; construct a camera visual frame with the camera's visual center as the center point, calculate the intersection-union ratio (IUR) between the camera visual frame and the object frame; query the first weight value corresponding to the Euclidean distance and the second weight value corresponding to the IUR, use the first weight value and the second weight value to perform a weighted calculation on the Euclidean distance and the IUR, and use the calculated result as the positional offset between the object frame and the camera visual center of the secondary camera.

[0173] In a specific application scenario, the directing module 404 is used to detect whether there is a projectable area in the image frame of the main video stream; when the projectable area is detected, a picture-in-picture area is formed using the projectable area, and the auxiliary video stream is projected into the picture-in-picture close-up area in the main video stream for playback, thus obtaining the live broadcast screen; when the projectable area is not detected in the image frame of the main video stream, a target sub-window is selected in the image frame of the main video stream as the picture-in-picture area, and the auxiliary video stream is projected into the picture-in-picture close-up area in the main video stream for playback, thus obtaining the live broadcast screen.

[0174] In specific application scenarios, the broadcast control module 404 is used to identify polygonal regions of preset colors in the image frame of the main video stream using an image segmentation model, use the polygonal regions as the projectable regions, generate an image mask with the same area as the projectable regions, and superimpose the image mask on the projectable regions to form a picture-in-picture close-up region; or, identify augmented reality (AR) identifiers in the image frame of the main video stream, use the area covered by the AR identifiers as the projectable regions, generate an image mask with the same area as the projectable regions, and superimpose the image mask on the projectable regions to form a picture-in-picture close-up region.

[0175] In a specific application scenario, the directing module 404 is used to read the anchor coordinates and product coordinates from the main video stream; based on the anchor coordinates and product coordinates, divide a display area including the anchor and the product in the main video stream; set a specified number of sub-windows on the image screen of the main video stream, calculate the intersection-over-union ratio (IoU) between each sub-window and the display area; select the target sub-window with the smallest IoU with the display area and less than the IoU threshold from the specified number of sub-windows, and use the target sub-window as the picture-in-picture area.

[0176] In specific application scenarios, the directing module 404 is also used to identify products with product identification codes pasted on them in the main video stream when the object to be zoomed in on is a live streamer, generate a combined feature vector, and input the combined feature vector into a preset posture classifier. The combined feature vector is used to represent the positional relationship between the live streamer's hand key points and the product. If the recognition result output by the preset posture classifier indicates that the live streamer's hand is in contact with the product, then a display area including the live streamer and the product is divided in the main video stream according to the live streamer's coordinates and the product's coordinates. The main video stream is then cropped and scaled according to the display area to obtain the processed main video stream.

[0177] In a specific application scenario, the broadcast control module 404 is used to identify multiple key points of the anchor's body, obtain the coordinates of the product identification code in the main video stream; establish a human coordinate system with the top left vertex of the object frame as the origin, perform coordinate transformation on the product identification code coordinates, transform the image coordinate system where the product identification code coordinates are located to the human coordinate system, and normalize the transformed product identification code coordinates to obtain first normalized coordinates; perform coordinate transformation on the coordinates of multiple key points corresponding to the multiple key points, transform the image coordinate system where the multiple key point coordinates are located to the human coordinate system, and normalize the transformed multiple key point coordinates respectively to obtain multiple second normalized coordinates; combine the first normalized coordinates and the multiple second normalized coordinates to generate a feature vector and use the feature vector as a combined feature vector.

[0178] In specific application scenarios, the broadcast control module 404 is also used to output the live broadcast image to the viewer's end; when receiving a playback adjustment instruction from the viewer's end, it determines the content to be played indicated by the playback adjustment instruction; adjusts the live broadcast image according to the content to be played; and outputs the adjusted live broadcast image to the viewer's end.

[0179] In specific application scenarios, the directing module 404 is used to: when the content to be played is a main camera image, use the main video stream as the live broadcast screen and output the live broadcast screen to the viewer's end so that the viewer's end can play the image screen of the main video stream in full screen; when the content to be played is a close-up image, use the auxiliary video stream as the live broadcast screen and output the live broadcast screen to the viewer's end so that the viewer's end can play the image screen of the auxiliary video stream in full screen; when the content to be played is a split-screen image, create a split-screen screen, divide the split-screen screen into a first projection area and a second projection area, project the main video stream to the first projection area, project the auxiliary video stream to the second projection area, and use the projected split-screen screen as the live broadcast screen and output the live broadcast screen to the viewer's end so that the viewer's end can play the projected split-screen screen in full screen.

[0180] The apparatus provided in this application embodiment, in response to a live broadcast start command, performs real-time image acquisition of the current live broadcast scene based on a main camera and a secondary camera. It acquires the main video stream captured by the main camera, determines the object to be zoomed in on, and performs image recognition on the main video stream to determine the object frame coordinates of the object to be zoomed in on the main video stream. Based on the object frame coordinates, it adjusts the shooting angle of the secondary camera, controls the secondary camera to face the object to be zoomed in, and acquires an image of the object to be zoomed in, obtaining a secondary video stream. It determines a picture-in-picture close-up area in the image frame of the main video stream and projects the image frame of the secondary video stream onto the picture-in-picture close-up area to obtain the live broadcast image. Based on the acquisition of video images using dual cameras, it can not only capture panoramic images but also capture close-up shots of selected objects, providing users with live broadcast images from different angles. The content displayed in the live broadcast image is more diverse and provides multi-faceted live broadcast information, allowing viewers to learn more about the live broadcast content and gain a more detailed understanding of the live broadcast content.

[0181] It should be noted that other corresponding descriptions of the functional units involved in the live broadcast output device provided in this application embodiment can be found in the following references. Figure 1 , Figures 2A to 2C as well as Figures 3A to 3B The corresponding descriptions in [the document] will not be repeated here.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0185] In an exemplary embodiment, see Figure 5 The invention also provides a computer device including a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the live broadcast output method described in the above embodiments.

[0186] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the live broadcast output method.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0189] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0190] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0191] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method of outputting a live picture, characterized by The application comprises: in response to a live broadcast start instruction, performing real-time image acquisition on a current live broadcast scene based on a main camera and a sub-camera, and obtaining a main video stream acquired by the main camera; determining an object to be close-up, performing image recognition on the main video stream, and determining object frame coordinates of an object frame of the object to be close-up on the main video stream; adjusting a shooting angle of the sub-camera according to the object frame coordinates, controlling the sub-camera to face the object to be close-up and perform image acquisition on the object to be close-up, and obtaining a sub-video stream; wherein the adjusting of the shooting angle of the sub-camera according to the object frame coordinates, the controlling of the sub-camera to face the object to be close-up and perform image acquisition on the object to be close-up, and the obtaining of the sub-video stream comprise: taking the object frame coordinates as a reference, controlling the sub-camera to rotate in the direction of the object to be close-up, and adjusting the shooting angle of the sub-camera; while controlling the sub-camera to rotate, controlling the sub-camera to perform object landmark detection by using an object detection model, the object landmark being a human head or a commodity center point; when the object landmark is detected based on the sub-camera, continuously obtaining landmark coordinates of the object landmark in an image currently acquired by the sub-camera; taking the landmark coordinates as a reference, continuously calculating a position offset between the object frame and a camera visual center of the sub-camera; continuing to control the sub-camera to rotate according to the position offset until the position offset is less than an offset threshold, and stopping the rotation, so that the sub-camera faces the object to be close-up; controlling the sub-camera to perform image acquisition on a currently faced direction, and obtaining the sub-video stream; determining a picture-in-picture close-up area in an image frame of the main video stream, projecting an image frame of the sub-video stream to the picture-in-picture close-up area, and obtaining a live broadcast picture, wherein when the live broadcast picture is generated, it is detected whether there is a projectable area in the image frame of the main video stream; when it is detected that the projectable area exists in the image frame of the main video stream, a picture-in-picture area is formed by using the projectable area, the sub-video stream is projected in the picture-in-picture close-up area in the main video stream for playing, and the live broadcast picture is obtained, wherein the projectable area is determined according to a polygonal area of a preset color in the image frame of the main video stream or according to an area covered by an AR identification code in the image frame of the main video stream; when it is detected that the projectable area does not exist in the image frame of the main video stream, a target sub-window is selected as the picture-in-picture area in the image frame of the main video stream, the sub-video stream is projected in the picture-in-picture close-up area in the main video stream for playing, and the live broadcast picture is obtained, wherein the target sub-window is determined according to a to-be-displayed area divided based on a host anchor coordinate of a host anchor in the main video stream and a commodity coordinate of a commodity.

2. The method of claim 1, wherein, The determining the object to be close-up and the image recognition of the main video stream to determine the object frame coordinates of the object frame of the object to be close-up on the main video stream, comprising: Obtaining a preset live broadcast parameter, and specifying an object to be close-up at a live broadcast time specified by the live broadcast parameter as the object to be close-up, the object to be close-up being any one of a host or a commodity; When the object to be close-up is the host, performing human body recognition on the host to obtain the object frame, and reading the coordinates of the object frame on the main video stream as the object frame coordinates; When the object to be close-up is the commodity, constructing the object frame according to the contour of the commodity, and reading the coordinates of the object frame on the main video stream as the object frame coordinates.

3. The method of claim 2, wherein, The human body recognition of the host to obtain the object frame, comprising: Down-sampling the main video stream to obtain the main video stream after down-sampling processing; Performing image recognition on the processed main video stream by using a human body detection model to determine multiple human body key points and the object frame of the object to be close-up in the main video stream.

4. The method of claim 2, wherein, The construction of the object frame according to the contour of the commodity, comprising: Down-sampling the main video stream to obtain the main video stream after down-sampling processing; Identifying a commodity in the processed main video stream, taking the commodity as the object to be close-up, the commodity being pasted with a commodity identification code or being consistent with a commodity picture uploaded by the host; Identifying the contour of the object to be close-up, and constructing the object frame of the object to be close-up according to the contour.

5. The method of claim 1, wherein, The position offset between the object frame and the camera visual center of the secondary camera is continuously calculated based on the landmark point coordinates, comprising: Reading the visual center coordinates of the camera visual center, and calculating the Euclidean distance between the landmark point coordinates and the visual center coordinates; Constructing a camera visual frame with the camera visual center as a center point, and calculating the intersection-over-union between the camera visual frame and the object frame; Querying a first weight value corresponding to the Euclidean distance and a second weight value corresponding to the intersection-over-union, and performing weighted calculation on the Euclidean distance and the intersection-over-union by using the first weight value and the second weight value, and taking the obtained calculation result as the position offset between the object frame and the camera visual center of the secondary camera.

6. The method of claim 1, wherein, The projection area is formed into a picture-in-picture area, comprising: Identifying a polygonal area of a preset color in an image frame of the main video stream by using an image segmentation model, taking the polygonal area as the projection area, generating an image mask with an area consistent with that of the projection area, and superimposing the image mask on the projection area to form a picture-in-picture close-up area; or Identifying an augmented reality (AR) identification code in an image frame of the main video stream, taking an area covered by the AR identification code as the projection area, generating an image mask with an area consistent with that of the projection area, and superimposing the image mask on the projection area to form a picture-in-picture close-up area.

7. The method of claim 1, wherein, The selecting a target sub-window as the picture-in-picture region in the image frame of the main video stream comprises: reading anchor coordinates of an anchor and product coordinates of a product in the main video stream; dividing a to-be-displayed region including the anchor and the product in the main video stream according to the anchor coordinates and the product coordinates; setting a specified number of sub-windows on the image frame of the main video stream, and respectively calculating an intersection-over-union between each sub-window and the to-be-displayed region; selecting a target sub-window with the smallest intersection-over-union with the to-be-displayed region and less than an intersection-over-union threshold from the specified number of sub-windows, and taking the target sub-window as the picture-in-picture region.

8. The method of claim 1, wherein, The method further comprises: when the to-be-close-up object is the anchor, identifying a product with a product identification code in the main video stream, generating a combined feature vector, and inputting the combined feature vector into a preset pose classifier, the combined feature vector being used to represent a position relationship between hand key points of the anchor and the product; if an identification result output by the preset pose classifier indicates that the hand of the anchor is in contact with the product, dividing a to-be-displayed region including the anchor and the product in the main video stream according to the anchor coordinates and the product coordinates; cropping and scaling the main video stream according to the to-be-displayed region to obtain a processed main video stream.

9. The method of claim 8, wherein, The generating a combined feature vector comprises: identifying a plurality of human key points of the anchor, and obtaining product identification code coordinates of the product identification code in the main video stream; establishing a human coordinate system with a top-left vertex of the object frame as an origin, performing coordinate conversion on the product identification code coordinates, transforming an image coordinate system in which the product identification code coordinates are located to the human coordinate system, and performing normalization processing on the transformed product identification code coordinates to obtain first normalized coordinates; performing coordinate conversion on a plurality of key point coordinates corresponding to the plurality of human key points, transforming an image coordinate system in which the plurality of key point coordinates are located to the human coordinate system, and respectively performing normalization processing on the transformed plurality of key point coordinates to obtain a plurality of second normalized coordinates; combining the first normalized coordinates and the plurality of second normalized coordinates to generate a feature vector, and taking the feature vector as a combined feature vector.

10. The method of claim 1, wherein, The method further comprises: outputting the live picture to a viewer end; when a play adjustment instruction of the viewer end is received, determining to-be-played content indicated by the play adjustment instruction; adjusting the live picture according to the to-be-played content, and outputting the adjusted live picture to the viewer end.

11. The method of claim 10, wherein, The adjusting the live picture according to the to-be-played content and outputting the adjusted live picture to the viewer end comprises: when the to-be-played content is a main camera image, taking the main video stream as the live picture, and outputting the live picture to the viewer end, so that the viewer end plays an image frame of the main video stream in full screen. When the to-be-played content is a close-up image, the auxiliary video stream is taken as the live picture, and the live picture is output to the audience terminal, so that the audience terminal plays the image picture of the auxiliary video stream in full screen; When the to-be-played content is a split-screen image, a split-screen picture is created, a first projection area and a second projection area are divided on the split-screen picture, the main video stream is projected to the first projection area, the auxiliary video stream is projected to the second projection area, and the projected split-screen picture is taken as the live picture, and the live picture is output to the audience terminal, so that the audience terminal plays the projected split-screen picture in full screen.

12. An output device for live picture, characterized in that Comprise: The acquisition module is configured to, in response to a live start instruction, perform real-time image acquisition on a current live scene based on a main camera and a secondary camera, and acquire a main video stream collected by the main camera; The intelligent identification module is configured to determine a to-be-close-up object and perform image identification on the main video stream to determine object frame coordinates of an object frame of the to-be-close-up object on the main video stream; The adjustment module is configured to adjust a shooting angle of the secondary camera according to the object frame coordinates, control the secondary camera to face the to-be-close-up object and perform image acquisition on the to-be-close-up object to obtain an auxiliary video stream; wherein the adjustment module is configured to control the secondary camera to rotate in a direction of the to-be-close-up object to adjust the shooting angle of the secondary camera based on the object frame coordinates; while controlling the secondary camera to rotate, the adjustment module controls the secondary camera to perform object landmark detection using an object detection model, the object landmark being a human head or a commodity center point; when the object landmark is detected based on the secondary camera, the adjustment module continuously acquires landmark coordinates of the object landmark in an image currently collected by the secondary camera; the adjustment module continuously calculates a position offset between the object frame and a camera visual center of the secondary camera based on the landmark coordinates; the adjustment module continues to control the secondary camera to rotate according to the position offset until the position offset is less than an offset threshold, and stops rotating, so that the secondary camera faces the to-be-close-up object; the adjustment module controls the secondary camera to perform image acquisition on a currently faced direction to obtain the auxiliary video stream; and The output module is configured to, when the to-be-played content is a close-up image, take the auxiliary video stream as the live picture, and output the live picture to the audience terminal, so that the audience terminal plays the image picture of the auxiliary video stream in full screen; or when the to-be-played content is a split-screen image, create a split-screen picture, divide a first projection area and a second projection area on the split-screen picture, project the main video stream to the first projection area, project the auxiliary video stream to the second projection area, take the projected split-screen picture as the live picture, and output the live picture to the audience terminal, so that the audience terminal plays the projected split-screen picture in full screen. The director module is configured to determine a picture-in-picture close-up area in an image frame of the main video stream, project an image frame of the auxiliary video stream to the picture-in-picture close-up area, and obtain a live picture. The director module is configured to detect whether a projectable area exists in the image frame of the main video stream when the live picture is generated. When the projectable area is detected in the image frame of the main video stream, a picture-in-picture area is formed by using the projectable area, the auxiliary video stream is projected in the picture-in-picture close-up area in the main video stream for playing, and the live picture is obtained. The projectable area is determined according to a polygonal area of a preset color in the image frame of the main video stream or according to an area covered by an AR identification code in the image frame of the main video stream. When the projectable area is not detected in the image frame of the main video stream, a target sub-window is selected as the picture-in-picture area in the image frame of the main video stream, the auxiliary video stream is projected in the picture-in-picture close-up area in the main video stream for playing, and the live picture is obtained. The target sub-window is determined according to a to-be-displayed area divided based on a host anchor coordinate of a host anchor and a commodity coordinate of a commodity in the main video stream.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

14. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 11.

Images