Adjustment method based on live broadcast picture and related device

Abstract

The invention discloses an adjustment method based on a live broadcast picture and a related device. The method comprises the following steps: for picture content of a first live broadcast picture, carrying out object detection to obtain a plurality of candidate detection frames corresponding to a plurality of preset objects in the first live broadcast picture; for a target detection frame in the plurality of candidate detection frames, segmenting a target object in the target detection frame from the first live broadcast picture to obtain a second live broadcast picture and a first object image of the target object; adjusting the first object image to obtain a second object image according to a preset adjustment strategy for highlighting one or more of a plurality of preset objects; and fusing the second object image and the second live broadcast picture to obtain a third live broadcast picture. According to the method, on the premise that the size of the live broadcast picture is not changed, the display effect of the preset object in the live broadcast picture is highlighted by detecting, segmenting and adjusting the target object in the live broadcast picture and fusing the target object to the live broadcast picture again, audience users are attracted to pay attention to the preset object in the live broadcast picture, and the live broadcast effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an adjustment method based on a live broadcast screen and a related device. Background Art

[0002] With the rapid development of live broadcast technology, live broadcast technology is everywhere in daily life or work. The anchor user can use the live broadcast software of the anchor terminal to shoot the live broadcast screen and send the live broadcast screen to the server, so that the server sends the live broadcast screen to the audience terminal for the audience user to watch the live broadcast screen.

[0003] In the related art, the method of adjusting the live broadcast screen includes extracting the outline image of the anchor from the live broadcast screen and beautifying the outline image of the anchor to make the anchor look better; or, replacing the live broadcast background other than the anchor in the live broadcast screen to make the live broadcast background more interesting.

[0004] However, the above methods only beautify the main objects such as the anchor in the live broadcast screen, or replace the background and other areas in the live broadcast screen, but cannot highlight the preset objects such as the anchor or objects in the live broadcast screen, resulting in failure to attract audience users to focus on the preset objects in the live broadcast screen, thereby affecting the live broadcast effect. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides an adjustment method and related devices based on a live broadcast screen, which can achieve the display effect of highlighting the preset object in the live broadcast screen by detecting, segmenting, adjusting the target object in the live broadcast screen and reintegrating it into the live broadcast screen without changing the size of the live broadcast screen, so as to attract the audience to focus on the preset object in the live broadcast screen, thereby improving the live broadcast effect.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides an adjustment method based on a live broadcast screen, the method comprising:

[0008] Performing object detection on the screen content of the first live screen to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen;

[0009] According to the target detection frames in the multiple candidate detection frames, segmenting the target object in the target detection frame from the first live picture, and obtaining a second live picture and a first object image of the target object;

[0010] Adjust the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of the multiple preset objects;

[0011] The second object image and the second live broadcast picture are fused to obtain a third live broadcast picture.

[0012] On the other hand, an embodiment of the present application provides an adjustment device based on a live picture, the device comprising: a detection unit, a segmentation unit, an adjustment unit and a fusion unit;

[0013] The detection unit is used to perform object detection on the picture content of the first live picture to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture;

[0014] The segmentation unit is used to segment the target object in the target detection frame from the first live picture according to the target detection frames in the multiple candidate detection frames, and obtain a second live picture and a first object image of the target object;

[0015] The adjustment unit is used to adjust the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of the multiple preset objects;

[0016] The fusion unit is used to perform image fusion on the second object image and the second live broadcast picture to obtain a third live broadcast picture.

[0017] Optionally, the preset adjustment strategy includes one or more of a size adjustment strategy and a position adjustment strategy.

[0018] Optionally, when the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is a size upsampling strategy, and the adjustment unit is specifically configured to:

[0019] The first object image is enlarged according to the size upsampling strategy to obtain the second object image; or,

[0020] The size adjustment strategy is a size downsampling strategy, and the adjustment unit is specifically used to:

[0021] The first object image is downsized according to the size downsampling strategy to obtain the second object image.

[0022] Optionally, when the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is a size enlargement model, and the adjustment unit is specifically configured to:

[0023] The first object image is enlarged according to the size enlargement model to obtain the second object image; the size enlargement model is used to enlarge the size of the first object image according to the mapping relationship between the first sample image and the second sample image, and the resolution of the first sample image is smaller than the resolution of the second sample image; or,

[0024] The size adjustment strategy is a size reduction model, and the adjustment unit is specifically used to:

[0025] The first object image is reduced in size according to the size reduction model to obtain the second object image; the size reduction model is used to reduce the size of the first object image according to a mapping relationship between a third sample image and a fourth sample image, and a resolution of the third sample image is greater than a resolution of the fourth sample image.

[0026] Optionally, when the preset adjustment strategy is the position adjustment strategy, the adjustment unit is specifically configured to:

[0027] The first object image is moved according to the position adjustment strategy to obtain the second object image; the position of the second object image is different from the position of the first object image.

[0028] Optionally, the fusion unit is specifically used for:

[0029] If it is determined according to the second object image and the second live picture that the second live picture includes a to-be-filled area, a similar area around the to-be-filled area in the second live picture is determined as a reference filling area;

[0030] Filling the to-be-filled area in the second live picture with content according to the area content of the reference filling area to obtain a fourth live picture;

[0031] The second object image and the fourth live broadcast picture are fused to obtain the third live broadcast picture.

[0032] Optionally, the device further comprises: a smoothing unit;

[0033] The smoothing unit is used to perform a smoothing process on the filled area to be filled in the fourth live picture according to the second live picture to obtain a fifth live picture;

[0034] The fusion unit is specifically used for:

[0035] The second object image and the fifth live broadcast picture are subjected to image fusion to obtain the third live broadcast picture.

[0036] Optionally, the smoothing unit is further used for:

[0037] According to the second live picture in the third live picture, the second object image in the third live picture is smoothed to obtain a sixth live picture.

[0038] Optionally, the smoothing unit is specifically used for:

[0039] Acquire a first gradient field of the second object image in the third live picture, and a second gradient field of the second live picture in the third live picture;

[0040] The second object image in the third live picture is smoothed according to the first gradient field and the second gradient field to obtain the sixth live picture.

[0041] Optionally, the smoothing unit is specifically used for:

[0042] The second object image in the third live picture is smoothed by an image smoothing model according to a mapping relationship between the second object image and the second live picture to obtain the sixth live picture.

[0043] Optionally, the detection unit is specifically used to:

[0044] Object detection is performed on the picture content of the first live picture according to the target frequency to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture.

[0045] Optionally, the device further includes: an updating unit;

[0046] The updating unit is specifically used for:

[0047] If the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the ith frame, the target frequency is updated according to the maximum frequency, where i is a positive integer;

[0048] If the multiple preset objects in the first live broadcast picture of the i+1th frame do not move relative to the multiple preset objects in the first live broadcast picture of the i-th frame, the target frequency is updated according to the difference frequency between the target frequency and the preset frequency; the updated target frequency is greater than or equal to the minimum frequency.

[0049] Optionally, the device further comprises: a determination unit;

[0050] The determining unit is specifically configured to:

[0051] Acquire multiple coordinate differences between multiple pixel coordinates of the first object in the (i+1)th frame of the first live picture and multiple pixel coordinates of the first object in the (i)th frame of the first live picture;

[0052] If a preset number of coordinate differences among the multiple coordinate differences is greater than a preset difference, it is determined that the first object in the (i+1)th frame of the first live picture moves relative to the first object in the (i)th frame of the first live picture.

[0053] Optionally, the preset adjustment strategy further includes a replacement adjustment strategy; and the adjustment unit is specifically configured to:

[0054] The first object image is replaced according to a preset replacement image using the replacement adjustment strategy to obtain the second object image.

[0055] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0056] The memory is used to store a computer program and transmit the computer program to the processor;

[0057] The processor is configured to execute the method described in any one of the preceding aspects according to instructions in the computer program.

[0058] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program runs on a computer device, the computer device executes the method described in any one of the above aspects.

[0059] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which, when executed on a computer device, enables the computer device to execute the method described in any one of the aforementioned aspects.

[0060] It can be seen from the above technical solution that, first, for the screen content of the first live screen, object detection is performed to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen; for the target detection frame in the multiple candidate detection frames, the target object in the target detection frame is segmented from the first live screen to obtain the second live screen and the first object image of the target object; this step can accurately segment the first live screen into the first object image of the target object to be adjusted and the second live screen to be fused by means of object detection and image segmentation. Then, according to the preset adjustment strategy for highlighting one or more of the multiple preset objects, the first object image is adjusted to obtain the second object image; the second object image and the second live screen are fused to obtain the third live screen; this step changes the first object image of the target object into the second object image by means of image adjustment and image fusion based on the preset adjustment strategy, and fuses it to the second live screen to obtain the third live screen, so that the third live screen can highlight one or more of the multiple preset objects. Based on this, this method can achieve the display effect of highlighting the preset objects in the live screen by detecting, segmenting, adjusting the target objects in the live screen and reintegrating them into the live screen without changing the size of the live screen, so as to attract the audience to focus on the preset objects in the live screen and thus improve the live broadcast effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technical members in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 A schematic diagram of a system architecture of a live screen adjustment method provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of an adjustment method based on a live screen provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of multiple candidate detection frames corresponding to multiple preset objects in a first live broadcast screen provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of a target detection frame in a plurality of candidate detection frames in a first live broadcast picture provided by an embodiment of the present application;

[0066] Figure 5 A schematic diagram of a second live broadcast screen and a first object image of a target object provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of adjusting a first object image of a target object to a second object image provided by an embodiment of the present application;

[0068] Figure 7 A schematic diagram of a third live broadcast screen provided in an embodiment of the present application;

[0069] Figure 8 A specific step diagram of an adjustment method based on a live screen provided in an embodiment of the present application;

[0070] Fig. 9 A schematic diagram of moving a first object image of a target object in a first live broadcast screen to a second object image provided by an embodiment of the present application;

[0071] Fig.10 A schematic diagram of a second live broadcast screen including an area to be filled provided in an embodiment of the present application;

[0072] Fig.11 A flow chart for determining a target frequency provided in an embodiment of the present application;

[0073] Fig.12 A structural diagram of an adjustment device based on a live screen provided in an embodiment of the present application;

[0074] Fig.13 A structural diagram of a terminal provided in an embodiment of the present application;

[0075] Fig.14 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0077] At present, the method for adjusting the live broadcast screen can be to extract the outline image of the anchor from the live broadcast screen and beautify the outline image of the anchor to obtain a better-looking anchor; or, to replace the live broadcast background except the outline image of the anchor in the live broadcast screen to obtain a more interesting live broadcast background.

[0078] However, after research, it was found that the above methods only beautify the main objects such as the anchor in the live broadcast screen, or replace the background and other areas in the live broadcast screen, but cannot highlight the preset objects such as the anchor or objects in the live broadcast screen, resulting in failure to attract audience users to focus on the preset objects in the live broadcast screen, thus affecting the live broadcast effect.

[0079] The embodiment of the present application provides an adjustment method based on a live broadcast screen. Without changing the size of the live broadcast screen, the method can detect, segment, adjust the target object in the live broadcast screen and reintegrate it into the live broadcast screen to achieve a display effect of highlighting the preset object in the live broadcast screen, so as to attract the audience to focus on the preset object in the live broadcast screen, thereby improving the live broadcast effect.

[0080] Next, the system architecture of the adjustment method based on the live screen will be introduced. Figure 1 , Figure 1 A schematic diagram of a system architecture of an adjustment method based on a live screen provided in an embodiment of the present application, wherein the system architecture includes a terminal 100, and the terminal 100 is used to execute the adjustment method based on a live screen.

[0081] The terminal 100 performs object detection on the picture content of the first live picture, and obtains multiple candidate detection frames corresponding to multiple preset objects in the first live picture.

[0082] As an example, the first live screen is live screen 1, and the multiple preset objects in live screen 1 include M objects, M is a positive integer, then the terminal 100 can perform object detection on the screen content of live screen 1 to obtain M candidate detection boxes corresponding to the M objects in live screen 1.

[0083] The terminal 100 segments a target object in the target detection frame from the first live picture according to the target detection frames in the plurality of candidate detection frames, and obtains a second live picture and a first object image of the target object.

[0084] As an example, based on the above example, the target detection frame among the M candidate detection frames is the i-th detection frame, i∈M, and the terminal 100 segments the target object in the i-th detection frame from the live broadcast screen 1 as the i-th object for the i-th detection frame, and obtains the second live broadcast screen as the live broadcast screen 2 and the first object image of the i-th object as the object image 1; wherein, the live broadcast screen 2 refers to the live broadcast screen 1 from which the object image 1 is segmented, and the live broadcast screen 2 includes M-1 preset objects among the M objects except the i-th object.

[0085] The terminal 100 adjusts the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of a plurality of preset objects.

[0086] As an example, the preset adjustment strategy includes one or more of a size adjustment strategy and a position adjustment strategy. Based on the above example, the terminal 100 adjusts the object image 1 according to one or more of the size adjustment strategy and the position adjustment strategy to obtain the second object image as the object image 2.

[0087] The terminal 100 performs image fusion on the second object image and the second live picture to obtain a third live picture.

[0088] As an example, based on the above example, the object image 2 and the live screen 2 are fused to obtain a third live screen as the live screen 3, which includes M-1 preset objects and the i-th object shown in the object image 2 (adjusted object image 1), for a total of M objects.

[0089] That is to say, through object detection and image segmentation, the first live screen can be accurately segmented into the first object image of the target object to be adjusted and the second live screen to be merged; through image adjustment and image fusion based on a preset adjustment strategy, the first object image of the target object is changed into the second object image, and merged into the second live screen to obtain a third live screen, so that the third live screen can highlight one or more of the multiple preset objects. Based on this, the method can achieve the display effect of highlighting the preset object in the live screen by detecting, segmenting, adjusting the target object in the live screen and re-merging it into the live screen without changing the size of the live screen, so as to attract the audience to focus on the preset object in the live screen, thereby improving the live broadcast effect.

[0090] It should be noted that in the embodiments of the present application, the computer device may be a server or a terminal, and the method provided in the embodiments of the present application may be executed by the terminal or the server alone, or by the terminal and the server in cooperation. Figure 1 The corresponding embodiments are mainly introduced by taking the terminal executing the method provided in the embodiments of the present application as an example.

[0091] In addition, when the method provided in the embodiment of the present application is executed by a server alone, its execution method is the same as Figure 1 The corresponding embodiments are similar, mainly replacing the terminal with a server. In addition, when the terminal and the server cooperate to execute the method provided in the embodiment of the present application, the steps that need to be reflected on the front-end interface can be executed by the terminal, and some steps that require background calculation and do not need to be reflected on the front-end interface can be executed by the server.

[0092] Among them, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a vehicle terminal or an aircraft, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services, but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. For example, the terminal and the server can be connected via a network, which can be a wired or wireless network.

[0093] The method provided in the embodiment of the present application involves artificial intelligence (AI) technology. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. AI is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0094] AI technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of AI generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. The software technology of AI mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The method provided in the embodiment of the present application mainly involves computer vision technology and machine learning / deep learning.

[0095] Computer Vision (CV) technology is a science that studies how to make machines "see". To put it more concretely, it refers to the use of cameras and computers to replace human eyes to identify, follow and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, CV studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of CV technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, and MAE can be quickly and widely applied to downstream specific tasks after fine tuning. The CV technology involved in the method provided in the embodiment of the present application mainly includes technologies such as image processing, image recognition, image semantic understanding, video processing, and video semantic understanding.

[0096] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. ML is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. ML and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by example. The pre-trained model is the latest development in deep learning, which integrates the above technologies.

[0097] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, audio and video, assisted driving, etc.

[0098] Next, the adjustment method based on the live screen provided by the embodiment of the present application will be described in detail by taking the terminal executing the method provided by the embodiment of the present application as an example and combining the accompanying drawings. Figure 2 , Figure 2 A flowchart of a method for adjusting a live screen based on an embodiment of the present application is provided, and the method includes:

[0099] S201: Perform object detection on the picture content of the first live picture to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture.

[0100] In the related art, the anchor's outline image can be extracted from the live broadcast screen and beautified to obtain a better-looking anchor; or the live broadcast background other than the anchor's outline image can be replaced to obtain a more interesting live broadcast background. However, after research, it was found that the above methods only beautify the anchor and other main objects in the live broadcast screen, or replace the background and other areas in the live broadcast screen, and cannot highlight the preset objects such as the anchor or objects in the live broadcast screen, resulting in the inability to attract audience users to focus on the preset objects in the live broadcast screen, thereby affecting the live broadcast effect.

[0101] Therefore, in the embodiment of the present application, in order to highlight the preset objects such as the anchor or the object in the live screen, it is first necessary to detect multiple preset objects in the live screen so that the preset objects can be highlighted in the live screen later. Based on this, the first live screen is obtained, and the object detection is performed on the screen content of the first live screen to obtain multiple candidate detection frames corresponding to the multiple preset objects in the first live screen.

[0102] In practical applications, S201 may be to perform object detection on the screen content of the first live screen through an object detection algorithm to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen. Wherein, the object detection algorithm may be the target detection algorithm Fast R-CNN algorithm, and the specific implementation process of S201 refers to: extracting features of the screen content of the first live screen through the Fast R-CNN algorithm to obtain image features of the first live screen; performing object-based region extraction on the screen content of the first live screen according to the image features of the first live screen to obtain multiple region images corresponding to multiple preset objects in the first live screen; performing object detection on multiple region images in the first live screen to obtain multiple predicted detection frames corresponding to multiple preset objects in the first live screen; performing regression processing on multiple predicted detection frames in the first live screen to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen.

[0103] The S201 processes the first live picture by object detection, and can accurately detect the candidate detection frame of each preset object in the first live picture, so as to provide effective and accurate detection data for subsequently accurately segmenting the first live picture into the first object image of the target object to be adjusted and the second live picture to be fused.

[0104] As an example of S201, see Figure 3 , Figure 3 A schematic diagram of multiple candidate detection frames corresponding to multiple preset objects in a first live broadcast screen provided by an embodiment of the present application, wherein: Figure 3 (a) indicates that the first live broadcast screen is live broadcast screen 1, and the multiple preset objects in live broadcast screen 1 include 4 objects, namely, the anchor, object 1, object 2 and object 3; Figure 3 (b) indicates that the terminal can detect the content of live screen 1 and obtain the candidate detection frame corresponding to the anchor in live screen 1 as the anchor detection frame, the candidate detection frame corresponding to object 1 as the object 1 detection frame, the candidate detection frame corresponding to object 2 as the object 2 detection frame, and the candidate detection frame corresponding to object 3 as the object 3 detection frame, that is, Figure 3 There are four dashed boxes in (b).

[0105] S202: Segment a target object in the target detection frame from the first live broadcast picture according to the target detection frame in the plurality of candidate detection frames, and obtain a second live broadcast picture and a first object image of the target object.

[0106] In an embodiment of the present application, in order to highlight preset objects such as hosts or objects in a live broadcast screen, after detecting multiple preset objects such as hosts and objects in the live broadcast screen, it is also necessary to select a target object from the multiple preset objects and segment the target object from the live broadcast screen, so that the preset object can be highlighted in the live broadcast screen by adjusting the target object later.

[0107] Based on this, after executing S201 to detect multiple candidate detection frames corresponding to multiple preset objects in the first live screen, it is also necessary to select a target detection frame for the multiple candidate detection frames, segment the target object in the target detection frame from the first live screen, and obtain the second live screen and the first object image of the target object. The second live screen refers to the first live screen from which the first object image of the target object is segmented, that is, the second live screen refers to other live screens in the first live screen except the first object image of the target object; the second live screen includes other objects in the multiple preset objects except the target object.

[0108] In practical applications, S202 may be in response to a selection operation for a target detection frame from a plurality of candidate detection frames, and determine a target detection frame from a plurality of candidate detection frames, or may be in response to detecting that the live voice corresponding to the first live screen includes a target object, and determine a target detection frame corresponding to the target object from a plurality of candidate detection frames; segment the target object in the target detection frame from the first live screen according to the target detection frame by an image segmentation algorithm, and obtain a second live screen and a first object image of the target object. Wherein, the image segmentation algorithm may be an instance segmentation algorithm Mask R-CNN algorithm, and the specific implementation process of S202 refers to: adding a segmentation branch for detecting the contour of the object on the basis of the Fast R-CNN algorithm, performing object-based contour detection according to the image features of the target detection frame and the target detection frame in the first live screen by the Mask R-CNN algorithm, and obtaining a target contour image of the target object; performing image segmentation on the first live screen according to the target contour image, and obtaining a target object image of the target object; performing image separation on the target object and the background area in the target object image, and obtaining a first object image of the target object.

[0109] The S202 processes the first live screen by image segmentation, and can accurately segment the first live screen into a first object image of the target object to be adjusted and a second live screen to be merged, so as to provide targeted and accurate image data for highlighting the preset object in the second live screen by subsequently adjusting the target object.

[0110] As an example of S202, based on the above S201 example, see Figure 4 , Figure 4A schematic diagram of a target detection frame in a plurality of candidate detection frames in a first live broadcast screen provided by an embodiment of the present application; Figure 3 Based on the Figure 4 In the live broadcast screen 1 shown in (a), the object 2 detection frame is selected from the host detection frame, the object 1 detection frame, the object 2 detection frame and the object 3 detection frame. The target detection frame is determined to be the object 2 detection frame from the host detection frame, the object 1 detection frame, the object 2 detection frame and the object 3 detection frame, that is, Figure 4 The bold box in (b) is shown.

[0111] See also Figure 5 , Figure 5 A schematic diagram of a second live broadcast screen and a first object image of a target object provided in an embodiment of the present application; in the above Figure 4 Based on this, the terminal segments object 2 from the live screen 1 and detects object 2 in the frame, and obtains Figure 5 The second live screen represented by (a) is live screen 2 and Figure 5 The first object image of the target object represented by (b) is object 2 image 1 of object 2; wherein live screen 2 refers to live screen 1 from which object 2 image 1 is segmented, that is, live screen 2 refers to other live screens in live screen 1 except object 2 image 1, and live screen 2 includes the anchor, object 1 and object 3.

[0112] S203: adjusting the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of a plurality of preset objects.

[0113] In an embodiment of the present application, in order to highlight preset objects such as an anchor or an object in a live broadcast screen, after segmenting a target object from multiple preset objects from the live broadcast screen, it is also necessary to adjust the target object based on the principle of highlighting one or more of the multiple preset objects so that the preset object can be highlighted in the live broadcast screen later.

[0114] Based on this, after executing S202 to divide the first live broadcast screen into the second live broadcast screen and the first object image of the target object, it is also necessary to adjust the first object image according to the preset adjustment strategy for highlighting one or more of the multiple preset objects to obtain the adjusted first object image as the second object image. The preset adjustment strategy for highlighting one or more of the multiple preset objects may be highlighting the target object among the multiple preset objects, or may be highlighting other objects among the multiple preset objects except the target object.

[0115] S203 processes the first object image of the target object by an image adjustment method based on a preset adjustment strategy, so that the first object image of the target object is accurately changed into a second object image. The second object image is integrated into the second live screen to highlight one or more of the multiple preset objects, thereby providing accurate and effective image data for subsequently highlighting the preset objects in the second live screen.

[0116] As an example of S203, based on the above S203 example, see Figure 6 , Figure 6 A schematic diagram of adjusting a first object image of a target object in a first live broadcast screen to a second object image is provided in an embodiment of the present application. Figure 5 Based on the above, according to the preset adjustment strategy for highlighting one or more of the plurality of preset objects, adjusting Figure 6 The object 2 image 1 represented by (a) is obtained Figure 6 The second object image represented by (b) is object 2 image 2.

[0117] S204: Perform image fusion on the second object image and the second live picture to obtain a third live picture.

[0118] In the embodiment of the present application, in order to achieve the display effect of highlighting the preset object in the live screen, so as to attract the audience to focus on the preset object in the live screen, after adjusting the target object, it is necessary to re-integrate the adjusted target object into the live screen, so as to highlight the preset object such as the anchor or object in the live screen. Based on this, after executing S203 according to the preset adjustment strategy for highlighting one or more of the multiple preset objects, adjusting the first object image of the target object to the second object image, it is necessary to integrate the second object image and the second live screen to obtain a third live screen.

[0119] The S204 processes the second live broadcast screen and the second object image by an image fusion method, and fuses the second object image obtained by changing the first object image of the target object into the second live broadcast screen to obtain a third live broadcast screen, so that the third live broadcast screen can highlight one or more of the multiple preset objects, thereby achieving a display effect of highlighting the preset objects in the third live broadcast screen, so as to attract audience users to focus on the preset objects in the third live broadcast screen, thereby improving the live broadcast effect.

[0120] As an example of S204, based on the above S203, see Figure 7 , Figure 7 A schematic diagram of a third live broadcast screen provided in an embodiment of the present application, in the above Figure 5 and Figure 6 Based on the integration Figure 6 (b) shows the object 2 image 2 and Figure 5 In the live broadcast screen 2 shown in (a), we get Figure 7 The third live broadcast screen represented is live broadcast screen 3, which includes the anchor, object 1 and object 3, as well as object 2 shown in object 2 image 2 (adjusted object 2 image 1).

[0121] In summary, see Figure 8 , Figure 8 A specific step diagram of an adjustment method based on a live screen provided in an embodiment of the present application, the specific steps include: a first step, object detection, that is, performing object detection on the screen content of a first live screen through an object detection algorithm, and obtaining multiple candidate detection frames corresponding to multiple preset objects in the first live screen; a second step, object selection, that is, in response to a selection operation on a target detection frame in multiple candidate detection frames, determining a target detection frame from multiple candidate detection frames; a third step, image segmentation, that is, segmenting a target object in a target detection frame from the first live screen according to the target detection frame through an image segmentation algorithm, and obtaining a second live screen and a first object image of the target object; a fourth step, image adjustment, that is, adjusting the first object image according to a preset adjustment strategy for highlighting one or more of the multiple preset objects to obtain a second object image; a fifth step, image fusion, that is, performing image fusion on the second object image and the second live screen to obtain a third live screen.

[0122] It can be seen from the above technical solution that, first, for the screen content of the first live screen, object detection is performed to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen; for the target detection frame in the multiple candidate detection frames, the target object in the target detection frame is segmented from the first live screen to obtain the second live screen and the first object image of the target object; this step can accurately segment the first live screen into the first object image of the target object to be adjusted and the second live screen to be fused by means of object detection and image segmentation. Then, according to the preset adjustment strategy for highlighting one or more of the multiple preset objects, the first object image is adjusted to obtain the second object image; the second object image and the second live screen are fused to obtain the third live screen; this step changes the first object image of the target object into the second object image by means of image adjustment and image fusion based on the preset adjustment strategy, and fuses it to the second live screen to obtain the third live screen, so that the third live screen can highlight one or more of the multiple preset objects. Based on this, this method can achieve the display effect of highlighting the preset objects in the live screen by detecting, segmenting, adjusting the target objects in the live screen and reintegrating them into the live screen without changing the size of the live screen, so as to attract the audience to focus on the preset objects in the live screen and thus improve the live broadcast effect.

[0123] In the embodiment of the present application, based on the principle of highlighting one or more of the multiple preset objects when executing S203, the target object is adjusted so that the preset object is subsequently highlighted in the live broadcast screen. The size of the target object can be adjusted, the position of the target object can be adjusted, or the size and position of the target object can be adjusted so that the preset object is subsequently highlighted in the live broadcast screen; the preset adjustment strategy can be a size adjustment strategy, a position adjustment strategy, or a size adjustment strategy and a position adjustment strategy, etc. Therefore, the present application provides a possible implementation method, and the preset adjustment strategy includes one or more of a size adjustment strategy and a position adjustment strategy.

[0124] In the embodiment of the present application, when the above S203 is specifically implemented, when the preset adjustment strategy is a size adjustment strategy, the size adjustment strategy may be a size upsampling strategy, a size downsampling strategy, a size enlargement model, or a size reduction model; different size adjustment strategies correspond to different implementation methods of S203, as shown below:

[0125] One implementation method refers to: the size adjustment strategy is a size upsampling strategy. For the first object image of the target object, the size upsampling strategy can generate new pixel values ​​according to the adjacent pixel values ​​in the first object image, and the new pixel values ​​will be inserted into the gaps of the existing pixel values ​​of the first object image, increasing the number of pixel values ​​of the first object image to enlarge the size of the first object image and improve the resolution of the first object image, so as to obtain the enlarged first object image as the second object image. Therefore, the present application provides a possible implementation method. When the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is the size upsampling strategy, and S203 is specifically S2031 (not shown in the figure): enlarge the first object image according to the size upsampling strategy to obtain the second object image. Among them, the size of the second object image is larger than the size of the first object image, and the resolution of the second object image is larger than the resolution of the first object image.

[0126] Among them, the size upsampling strategy can be a bilinear interpolation algorithm or a bicubic interpolation algorithm. The bilinear interpolation algorithm refers to calculating one new pixel value through 4 (2×2) adjacent pixel values ​​in the image, and the bicubic interpolation algorithm refers to calculating one new pixel value through 16 (4×4) adjacent pixel values ​​in the image.

[0127] The S2031 processes the first object image of the target object through a size upsampling strategy, so that the first object image of the target object is accurately and controllably enlarged to a second object image, and the second object image is integrated into the second live screen to highlight the target object among multiple preset objects, thereby providing accurate and effective image data for subsequently highlighting the target object among multiple preset objects in the second live screen.

[0128] As an example of S2031, based on the above S202 example, a bilinear interpolation algorithm or a bicubic interpolation algorithm is used to increase the Figure 6 The second object image obtained from the object 2 image 1 represented by (a) is the object 2 image 3, the size of the object 2 image 3 is larger than the size of the object 2 image 1, and the resolution of the object 2 image 3 is greater than the resolution of the object 2 image 1.

[0129] Another implementation method refers to: the size adjustment strategy is a size downsampling strategy. For the first object image of the target object, the size downsampling strategy can generate new pixel values ​​according to the adjacent pixel values ​​in the first object image, and use the new pixel values ​​to replace the adjacent pixel values, thereby reducing the number of pixel values ​​of the first object image, so as to reduce the size of the first object image and reduce the resolution of the first object image, and obtain the reduced first object image as the second object image. Therefore, the present application provides a possible implementation method. When the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is the size downsampling strategy, and S203 is specifically S2032 (not shown in the figure): the first object image is reduced according to the size downsampling strategy to obtain the second object image. Among them, the size of the second object image is smaller than the size of the first object image, and the resolution of the second object image is smaller than the resolution of the first object image.

[0130] The downsampling strategy may also be a bilinear interpolation algorithm or a bicubic interpolation algorithm.

[0131] S2032 processes the first object image of the target object through a size downsampling strategy, so that the first object image of the target object is accurately and controllably reduced to a second object image, and the second object image is integrated into the second live broadcast screen to highlight other objects except the target object among multiple preset objects, thereby providing accurate and effective image data for subsequently highlighting other objects except the target object among multiple preset objects in the second live broadcast screen.

[0132] As an example of S2032, based on the above S203 example, the bilinear interpolation algorithm or the bicubic interpolation algorithm is used to reduce the Figure 6 The object 2 image 1 represented by (a) is obtained Figure 6 The second object image represented by (b) is object 2 image 2 , the size of object 2 image 2 is smaller than the size of object 2 image 1 , and the resolution of object 2 image 2 is smaller than the resolution of object 2 image 1 .

[0133] Another implementation method refers to: the size adjustment strategy is a size enlargement model, and the training process of the size enlargement model is to pre-acquire a large number of first sample images and second sample images, the resolution of the first sample image is smaller than the resolution of the corresponding second sample image, that is, the size of the first sample image is smaller than the size of the corresponding second sample image; based on the large number of first sample images and second sample images, train a first preset model to obtain a size enlargement model; the size enlargement model learns the mapping relationship between the first sample image and the second sample image, that is, the mapping relationship between the low-resolution image and the high-resolution image.

[0134] Based on this, for the first object image of the target object, the first object image is enlarged by the size enlargement model, that is, the size of the first object image is enlarged according to the mapping relationship between the first sample image and the second sample image, and the enlarged first object image is obtained as the second object image. Therefore, the present application provides a possible implementation method, when the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is the size enlargement model, S203 is specifically S2033 (not shown in the figure): the first object image is enlarged according to the size enlargement model to obtain the second object image; the size enlargement model is used to enlarge the size of the first object image according to the mapping relationship between the first sample image and the second sample image, and the resolution of the first sample image is smaller than the resolution of the second sample image. Among them, the size of the second object image is larger than the size of the first object image.

[0135] The size enlargement model may be a super resolution model (Super Resolution Convolutional Neural Network, SRCNN), which is used to convert an input low-resolution image into a high-resolution image.

[0136] S2033 processes the first object image of the target object through a size magnification model, so that the first object image of the target object is accurately and intelligently magnified into a second object image. The second object image is integrated into the second live screen to highlight the target object among multiple preset objects, thereby providing accurate and effective image data for subsequently highlighting the target object among multiple preset objects in the second live screen.

[0137] As an example of S2033, based on the above S202 example, SRCNN is used to amplify Figure 6 The second object image obtained from the object 2 image 1 represented by (a) is the object 2 image 4, the size of the object 2 image 4 is larger than the size of the object 2 image 1, and the resolution of the object 2 image 4 is larger than the resolution of the object 2 image 1.

[0138] Another implementation method refers to: the size adjustment strategy is a size reduction model, the training process of the size reduction model is to obtain a large number of third sample images and fourth sample images in advance, the resolution of the third sample image is greater than the resolution of the corresponding fourth sample image, that is, the size of the third sample image is greater than the size of the corresponding fourth sample image; based on a large number of third sample images and fourth sample images, the second preset model is trained to obtain a size reduction model; the size reduction model learns the mapping relationship between the third sample image and the fourth sample image, that is, the mapping relationship between the high-resolution image and the low-resolution image. Based on this, for the first object image of the target object, the first object image is reduced by the size reduction model, that is, the size of the first object image is reduced according to the mapping relationship between the third sample image and the fourth sample image, and the reduced first object image is obtained as the second object image. Therefore, the present application provides a possible implementation method. When the preset adjustment strategy is a size adjustment strategy, the size adjustment strategy is a size reduction model. S203 is specifically S2034 (not shown in the figure): the first object image is reduced according to the size reduction model to obtain the second object image; the size reduction model is used to reduce the size of the first object image according to the mapping relationship between the third sample image and the fourth sample image, and the resolution of the third sample image is greater than the resolution of the fourth sample image. Among them, the size of the second object image is smaller than the size of the first object image

[0139] The size reduction model may be an Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) model, which is used to convert an input high-resolution image into a low-resolution image.

[0140] The S2034 processes the first object image of the target object through a size reduction model, so that the first object image of the target object is accurately and intelligently reduced to a second object image, and the second object image is integrated into the second live screen to highlight other objects except the target object among multiple preset objects, thereby providing accurate and effective image data for subsequently highlighting other objects except the target object among multiple preset objects in the second live screen.

[0141] As an example of S2034, based on the above S203 example, ESRGAN is used to reduce Figure 6 The object 2 image 1 represented by (a) is obtained Figure 6 The second object image represented by (b) is object 2 image 2 , the size of object 2 image 2 is smaller than the size of object 2 image 1 , and the resolution of object 2 image 2 is smaller than the resolution of object 2 image 1 .

[0142] In the embodiment of the present application, when the above S203 is specifically implemented, when the preset adjustment strategy is a position adjustment strategy, for the first object image of the target object, the first object image is moved by the position adjustment strategy to obtain a second object image whose position is different from the position of the first object image. Therefore, the present application provides a possible implementation method, when the preset adjustment strategy is a position adjustment strategy, S203 is specifically S2035 (not shown in the figure): the first object image is moved according to the position adjustment strategy to obtain a second object image; the position of the second object image is different from the position of the first object image.

[0143] S2035 processes the first object image of the target object through a position adjustment strategy, so that the first object image of the target object is accurately moved to the second object image. The second object image is integrated into the second live screen to highlight the target object among multiple preset objects or other objects among multiple preset objects except the target object, thereby providing accurate and effective image data for subsequently highlighting the preset object in the second live screen.

[0144] As an example of S2035, based on the above S201 example, the terminal responds to Figure 4 In (a), the object 1 detection frame is selected from the host detection frame, the object 1 detection frame, the object 2 detection frame, and the object 3 detection frame in the live screen 1, and the target detection frame is determined to be the object 1 detection frame from the host detection frame, the object 1 detection frame, the object 2 detection frame, and the object 3 detection frame, and object 1 in the object 1 detection frame is segmented from the live screen 1, and the first object image of the target object is obtained as object 1 image 1 of object 1; see Fig. 9 , Fig. 9 A schematic diagram of a first object image of a target object in a first live screen being moved to a second object image provided by an embodiment of the present application, wherein the first object image of the target object in the first live screen is moved by a position adjustment strategy Fig. 9 The second object image represented by (a) is the object 1 image 1, and the position of the object 1 image 2, that is, the coordinates of the central pixel point (x t ,y t ), which is different from the position of the object 1 image 1, that is, the coordinate of the center pixel (x s ,y s ).

[0145] Based on the above description, S202-S203 can be: according to the target object in the live voice corresponding to the first live screen, determine the target detection frame corresponding to the target object from multiple candidate detection frames; segment the target object in the target detection frame from the first live screen according to the target detection frame through an image segmentation algorithm, and obtain the second live screen and the first object image of the target object; enlarge the first object image according to the size upsampling strategy to obtain the second object image; or, enlarge the first object image according to the size enlargement model to obtain the second object image; or, move the first object image according to the position adjustment strategy to obtain the second object image.

[0146] This method can automatically detect whether the live broadcast voice corresponding to the first live broadcast screen includes a target object among multiple preset objects. If so, the first live broadcast screen can be accurately divided into a first object image of the target object to be adjusted and a second live broadcast screen to be fused by processing the first live broadcast screen through an image segmentation method; the first object image of the target object is processed through a size upsampling strategy, so that the first object image of the target object is accurately and controllably enlarged to the second object image; or, the first object image of the target object is processed through a size magnification model, so that the first object image of the target object is accurately and intelligently enlarged to the second object image, or, the first object image of the target object is processed through a position adjustment strategy, so that the first object image of the target object is accurately moved to the second object image; the second object image is fused to the second live broadcast screen to highlight the target object among multiple preset objects, so that the target object among multiple preset objects can be highlighted in the second live broadcast screen subsequently.

[0147] That is, this method can automatically detect the target object among the multiple preset objects described by the live broadcast voice corresponding to the live broadcast screen for multiple preset objects in the live broadcast screen, so as to automatically enlarge the target object in the live broadcast screen or adjust the position of the target object in the live broadcast screen, thereby highlighting the target object in the live broadcast screen. In the product live broadcast scenario, the target object can be the target product being introduced in the live broadcast voice corresponding to the product live broadcast screen.

[0148] As an example of S202-S203, in the above Figure 3 On the basis of, in response to detecting that the live broadcast voice corresponding to the live broadcast picture 1 includes the target object being object 2, the terminal determines that the target detection frame is the object 2 detection frame from the anchor detection frame, the object 1 detection frame, the object 2 detection frame, and the object 3 detection frame, that is, Figure 4 In (b), the bold box; in the above Figure 4 Based on this, the terminal segments object 2 from the live screen 1 and detects object 2 in the frame, and obtains Figure 5 The second live screen represented by (a) is live screen 2 and Figure 5The first object image of the target object represented by (b) is the object 2 image 1 of the object 2. Figure 5 Based on the bilinear interpolation algorithm or the bicubic interpolation algorithm, the Figure 6 The second object image represented by (a) is the object 2 image 1, and the second object image is the object 2 image 3; or, by SRCNN amplification Figure 6 The second object image represented by (a) is the object 2 image 1, and the second object image is the object 2 image 4; or, by moving the position adjustment strategy Figure 6 The second object image obtained by the object 2 image 1 represented by (a) is the object 2 image 5, and the object 2 image 5 can be located at the center of the live broadcast screen 2. In the product live broadcast scene, the object 2 is the target product being introduced in the live broadcast voice corresponding to the live broadcast screen 1.

[0149] In the embodiment of the present application, when the above S203 is specifically implemented, when the preset adjustment strategy includes a size adjustment strategy and a position adjustment strategy, for the first object image of the target object, the first object image is scaled by the size adjustment strategy to obtain an intermediate object image whose size is different from the size of the first object image; the intermediate object image is moved by the position adjustment strategy to obtain a second object image whose position is different from the position of the intermediate object image. Therefore, the present application provides a possible implementation method, when the preset adjustment strategy is a position adjustment strategy, S203 includes the following S2036-S2037 (not shown in the figure):

[0150] S2036: scaling the first object image according to the size adjustment strategy to obtain an intermediate object image; the size of the intermediate object image is different from the size of the first object image.

[0151] S2037: Move the intermediate object image according to the position adjustment strategy to obtain a second object image; the position of the second object image is different from the position of the intermediate object image.

[0152] The examples of S2036-S2037 refer to the above examples and will not be repeated here.

[0153] In an embodiment of the present application, when the above-mentioned S204 is specifically implemented, since the second live broadcast screen refers to the first live broadcast screen from which the first object image of the target object is segmented, that is, the second live broadcast screen refers to other live broadcast screens in the first live broadcast screen except the first object image of the target object, and the second object image is the adjusted first object image; therefore, the first object image is adjusted to the second object image, for example, the first object image is reduced or shrunk to the second object image, or the first object image is moved to the second object image, and an area to be filled that does not match the surrounding pixels appears in the second live broadcast screen, and the area to be filled also needs to be filled with content that matches the surrounding pixels, so as to subsequently fuse the second object image and the filled second live broadcast screen to obtain a third live broadcast screen.

[0154] Based on this, when determining that the second live screen includes a to-be-filled area through the second object image and the second live screen, first, it is necessary to determine the surrounding similar area of ​​the to-be-filled area in the second live screen as a reference filling area; then, fill the to-be-filled area in the second live screen according to the area content of the reference filling area, and obtain the filled second live screen as the fourth live screen; finally, fuse the second object image and the fourth live screen to obtain the third live screen. Therefore, the present application provides a possible implementation method, S204 includes the following S2041-S2043 (not shown in the figure):

[0155] S2041: If it is determined according to the second object image and the second live picture that the second live picture includes a to-be-filled area, a similar area around the to-be-filled area in the second live picture is determined as a reference filling area.

[0156] S2042: Filling the area to be filled in the second live picture with content according to the area content of the reference filling area to obtain a fourth live picture.

[0157] In actual applications, filling the area to be filled in the second live screen according to the area content of the reference filling area, and obtaining the filled second live screen as the fourth live screen means: filling the texture, color, shape and other features of the area content of the reference filling area into the area to be filled in the second live screen, and obtaining the filled second live screen as the fourth live screen.

[0158] S2043: Perform image fusion on the second object image and the fourth live picture to obtain a third live picture.

[0159] When the second live screen includes an area to be filled, S2041-S2043 fills the area to be filled by using the area contents of similar areas around the area to be filled to obtain a fourth live screen where no area to be filled exists, thereby fusing the second object image to obtain a third live screen. This can avoid the appearance of an area to be filled that does not match the surrounding pixels in the third live screen, thereby improving the display effect of the third live screen.

[0160] As an example of S2041-S2043, see Fig.10 , Fig.10 A schematic diagram of a second live broadcast screen including an area to be filled provided in an embodiment of the present application, in which the above adjustment Figure 6 The object 2 image 1 represented by (a) is obtained Figure 6 Based on the object 2 image 2 represented by (b), the second live broadcast screen includes an area to be filled, that is, Fig.10 The area with the diagonal pattern is filled in the live screen 2. The similar area around the area to be filled in the live screen 2 is determined as the reference filling area; the area to be filled in the live screen 2 is filled according to the area content of the reference filling area, and the filled live screen 2 is obtained as the fourth live screen, that is, the live screen 4; the live screen 3 is obtained by fusing the object image 2 and the live screen 4.

[0161] In addition, in an embodiment of the present application, on the basis of the above S2041-S2042, in the fourth live picture obtained from the second live picture, there is an unnatural transition problem between the filled area to be filled and the surrounding pixels; in order to solve this problem, it is also necessary to smooth the filled area to be filled in the fourth live picture based on the second live picture, and obtain the smoothed fourth live picture as the fifth live picture; correspondingly, when S2043 is specifically implemented, it is necessary to fuse the second object image and the fifth live picture to obtain the third live picture. Therefore, the present application provides a possible implementation method, after executing the above S2041-S2042, the method also includes S1 (not shown in the figure): smoothing the filled area to be filled in the fourth live picture according to the second live picture to obtain the fifth live picture; correspondingly, S2043 is specifically S2044 (not shown in the figure): performing image fusion on the second object image and the fifth live picture to obtain the third live picture.

[0162] Among them, S1 can be smoothing the filled area to be filled in the fourth live picture according to the second live picture through the Boisson fusion algorithm to obtain the fifth live picture. The Boisson fusion algorithm is used to smooth the source image in the fused image according to the gradient field of the source image and the gradient field of the target image when fusing the source image to the target image.

[0163] S1 and S2044 obtain the fifth live picture by smoothing the filled area to be filled in the fourth live picture, so that the transition between the filled area to be filled in the fifth live picture and the surrounding pixels is more natural, and the second object image is integrated to obtain the third live picture, so as to avoid unnatural pixel transition in the third live picture, thereby further improving the display effect of the third live picture.

[0164] As an example of S1 and S2044, based on the live screen 4 obtained by the above S2041-S2043 examples, the filled area to be filled in the live screen 4 is smoothed based on the live screen 2 to obtain the smoothed live screen 4 as the fifth live screen, that is, the live screen 5; correspondingly, the object image 2 and the live screen 5 are fused to obtain the live screen 3.

[0165] In addition, in the embodiment of the present application, based on the above S204, in the third live picture obtained by fusing the second object image and the second live picture, there is an unnatural transition problem between the second object image and the second live picture; in order to solve this problem, it is also necessary to smooth the second object image in the third live picture based on the second live picture in the third live picture, and obtain the smoothed third live picture as the sixth live picture. Therefore, the present application provides a possible implementation method, and the method also includes S2 (not shown in the figure): according to the second live picture in the third live picture, the second object image in the third live picture is smoothed to obtain the sixth live picture.

[0166] The S2 smoothes the second object image in the third live screen to make the transition between the second object image in the sixth live screen and the second live screen more natural, and replaces the third live screen with the sixth live screen. On the basis of achieving the display effect of highlighting the preset object in the live screen so as to attract the audience to focus on the preset object in the live screen, the display effect of the sixth live screen is improved to further improve the live effect.

[0167] As an example of S2, based on the above S204 example, the image 2 of the object 2 in the live picture 3 is smoothed based on the live picture 2 in the live picture 3, and the smoothed live picture 3 is obtained as the sixth live picture, that is, the live picture 6.

[0168] The above S2 can be implemented in at least the following two ways:

[0169] One implementation method is: according to the second live screen in the third live screen, the second object image in the third live screen is smoothed by the Boisson fusion algorithm to obtain the sixth live screen; because the Boisson fusion algorithm is used to smooth the source image in the fused image according to the gradient field of the source image and the gradient field of the target image when fusing the source image to the target image; therefore, taking the second object image as the source image and the second live screen as the target image, first, it is necessary to obtain the first gradient field of the second object image in the third live screen and the second gradient field of the second live screen in the third live screen; then, based on the first gradient field and the second gradient field, the second object image in the third live screen is smoothed to obtain the smoothed third live screen as the sixth live screen. That is, the present application provides a possible implementation method, S2 includes the following S21-S22 (not shown in the figure):

[0170] S21: Acquire a first gradient field of a second object image in a third live picture, and a second gradient field of a second live picture in the third live picture.

[0171] S22: performing a smoothing process on the second object image in the third live picture according to the first gradient field and the second gradient field to obtain a sixth live picture.

[0172] As an example of S21-S22, on the basis of the above S2 example, the first gradient field of the image 2 of object 2 in the live screen 3 is obtained as gradient field 1, and the second gradient field of the live screen 2 in the live screen 3 is obtained as gradient field 2, and the image 2 of object 2 in the live screen 3 is smoothed based on the gradient field 1 and the gradient field 2, and the smoothed live screen 3 is obtained as the sixth live screen, that is, live screen 6.

[0173] Another implementation method refers to: obtaining a large number of source images and target images in advance, and the source image is to be fused to the target image; based on the large number of source images and target images, training a third preset model to obtain an image smoothing model; the image smoothing model learns the mapping relationship between the source image and the target image. Based on this, the second object image is used as the source image and the second live screen is used as the target image, and the second object image in the third live screen is smoothed according to the mapping relationship between the second object image and the second live screen through the image smoothing model, and the smoothed third live screen is obtained as the sixth live screen. Therefore, the present application provides a possible implementation method, S2 is specifically S23 (not shown in the figure): the second object image in the third live screen is smoothed according to the mapping relationship between the second object image and the second live screen through the image smoothing model to obtain the sixth live screen.

[0174] As an example of S23, based on the above S2 example, the image 2 of object 2 in the live screen 3 is smoothed according to the mapping relationship between the image 2 of object 2 in the live screen 3 and the live screen 2 in the live screen 3 through the image smoothing model, and the smoothed live screen 3 is obtained as the sixth live screen, that is, live screen 6.

[0175] In the embodiment of the present application, if the above adjustment method based on the live screen is executed in real time, more calculation steps need to be executed and more calculation resources are consumed; in order to save calculation resources, a certain detection frequency can be set as the target frequency, and according to the target frequency, the object detection is performed on the screen content of the first live screen to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen. Therefore, the present application provides a possible implementation method, S201 is specifically S2011 (not shown in the figure): according to the target frequency, the screen content of the first live screen is subjected to object detection, and multiple candidate detection frames corresponding to multiple preset objects in the first live screen are obtained.

[0176] The S2011 processes the first live picture by object detection according to the target frequency, and can intermittently and accurately detect the candidate detection frame of each preset object in the first live picture. On the basis of saving computing resources, it provides effective and accurate detection data for the subsequent accurate segmentation of the first live picture into the first object image of the target object to be adjusted and the second live picture to be fused.

[0177] As an example of S2011, the target frequency is tar_freq. Based on the above S201 example, according to tar_freq, for the picture content of live screen 1, object detection is performed to obtain the host detection frame, object 1 detection frame, object 2 detection frame and object 3 detection frame in live screen 1.

[0178] Among them, the target frequency can be dynamically updated based on whether the preset object in the first live screen of the previous and next two frames moves. In actual applications, any preset object among the multiple preset objects is taken as the first object, and i is a positive integer; when the first object in the first live screen of the i+1th frame moves relative to the first object in the first live screen of the i-th frame, it is necessary to perform extremely frequent object detection to obtain multiple candidate detection frames corresponding to the multiple preset objects in the first live screen, and the maximum frequency is taken as the target frequency; when the multiple preset objects in the first live screen of the i+1th frame do not move relative to the multiple preset objects in the first live screen of the i-th frame, the target frequency can be lowered, and there is no need to perform extremely frequent object detection to obtain multiple candidate detection frames corresponding to the multiple preset objects in the first live screen, and the difference frequency between the target frequency and the preset frequency is taken as the target frequency, and when the target frequency is the minimum frequency, there is no need to lower the target frequency. Therefore, the present application provides a possible implementation method, and the step of obtaining the target frequency includes the following S3 or S4:

[0179] S3: If the first object in the first live broadcast picture of the (i+1)th frame moves relative to the first object in the first live broadcast picture of the (i)th frame, update the target frequency according to the maximum frequency, where i is a positive integer.

[0180] S4: If the multiple preset objects in the first live broadcast picture of the i+1th frame do not move relative to the multiple preset objects in the first live broadcast picture of the ith frame, update the target frequency according to the difference frequency between the target frequency and the preset frequency; the updated target frequency is greater than or equal to the minimum frequency.

[0181] S3 and S4 dynamically update the target frequency based on whether the preset object in the first live screen of the two frames before and after moves, and on the basis of saving computing resources, when the preset object in the first live screen of the two frames before and after moves, the object detection is performed extremely frequently with the maximum detection frequency to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen; when the preset object in the first live screen of the two frames before and after does not move, the detection frequency of multiple candidate detection frames corresponding to multiple preset objects in the first live screen is reduced.

[0182] As an example of S3 and S4, based on the above S201 example, see Fig.11 , Fig.11 A flowchart for determining a target frequency is provided for an embodiment of the present application, and the process refers to: obtaining a target frequency; determining whether a first object in a first live broadcast picture of the i+1th frame moves relative to a first object in a first live broadcast picture of the i-th frame, and if so, updating the target frequency according to a maximum frequency; if not, that is, multiple preset objects in a first live broadcast picture of the i+1th frame do not move relative to multiple preset objects in a first live broadcast picture of the i-th frame, updating the target frequency according to a difference frequency between the target frequency and the preset frequency; the updated target frequency is greater than or equal to the minimum frequency.

[0183] Among them, the implementation method of determining that the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the i-th frame refers to: determining that the positions of a part of the pixel points of the first object in the first live broadcast picture of the i+1th frame have changed relative to the multiple pixel points of the first object in the first live broadcast picture of the i-th frame. Based on this, firstly, obtain a plurality of coordinate differences between the coordinates of the multiple pixel points of the first object in the first live broadcast picture of the i+1th frame and the coordinates of the multiple pixel points of the first object in the first live broadcast picture of the i-th frame; then, determine whether a preset number of coordinate differences among the plurality of coordinate differences is greater than the preset difference; if so, it can be determined that the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the i-th frame; if not, it indicates that the first object in the first live broadcast picture of the i+1th frame does not move relative to the first object in the first live broadcast picture of the i-th frame; wherein the preset number is configured according to actual needs. Therefore, the present application provides a possible implementation method, and the steps of determining that the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the i-th frame include the following S5-S6:

[0184] S5: Obtain multiple coordinate differences between multiple pixel coordinates of the first object in the (i+1)th frame of the first live picture and multiple pixel coordinates of the first object in the (i)th frame of the first live picture.

[0185] S6: If a preset number of coordinate differences among the plurality of coordinate differences is greater than a preset difference, it is determined that the first object in the (i+1)th frame of the first live picture moves relative to the first object in the (i)th frame of the first live picture.

[0186] In S5-S6, the positions of a part of pixel points of the first object in the first live broadcast picture of the i+1th frame relative to the corresponding part of pixel points of the first object in the first live broadcast picture of the i-th frame change, and it is determined that the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the i-th frame. It can accurately judge whether the preset object in the first live broadcast pictures of the previous and next two frames moves, and provide an accurate update basis for the subsequent dynamic update of the target frequency.

[0187] As an example of S5-S6, the preset difference is Diff_Threshold, and the N coordinate differences between the N pixel coordinates of the first object in the first live picture of the i+1th frame and the N pixel coordinates of the first object in the first live picture of the i-th frame are obtained, and the coordinate difference refers to the spatial distance between the pixel coordinates of the first object in the first live picture of the i+1th frame and the pixel coordinates of the first object in the first live picture of the i-th frame; it is determined whether N / 16 coordinate differences among the N coordinate differences are greater than Diff_Threshold, where N is a positive integer and N is a multiple of 16. If so, it can be determined that the first object in the first live picture of the i+1th frame moves relative to the first object in the first live picture of the i-th frame. Among them, Diff_Threshold can be 1 / 100 of the width of the first live picture.

[0188] In addition, in an embodiment of the present application, when executing S203, based on the principle of highlighting one or more of a plurality of preset objects, the target object is adjusted so that the preset object can be highlighted in the live broadcast screen later. It can also be to replace the target object to highlight the preset object in the live broadcast screen. Then the preset adjustment strategy can also be a replacement adjustment strategy. For the first object image of the target object, the first object image is replaced with a preset replacement image different from the first object image through the replacement adjustment strategy, and the preset replacement image is used as the second object image. Therefore, the present application provides a possible implementation method, the preset adjustment strategy also includes a replacement adjustment strategy, S203 is specifically S2038 (not shown in the figure): the first object image is replaced according to the preset replacement image through the replacement adjustment strategy to obtain the second object image.

[0189] S2038 processes the first object image of the target object through a replacement adjustment strategy, so that the first object image of the target object is replaced with a preset replacement image as the second object image. The second object image is integrated into the second live screen to highlight other objects except the target object among multiple preset objects, thereby providing accurate and effective image data for subsequently highlighting other objects except the target object among multiple preset objects in the second live screen.

[0190] As an example of S2038, based on the above S203 example, the adjustment strategy is replaced by Figure 6 The object 2 image 1 represented by (a) is replaced with a preset replacement image, and the second object image obtained is the preset replacement image.

[0191] In addition, in an embodiment of the present application, after executing S202 to divide the first live broadcast screen into the second live broadcast screen and the first object image of the target object, the first object image can also be deleted, and the second live broadcast screen can be filled with content to obtain a seventh live broadcast screen; so that other objects except the target object among multiple preset objects can be highlighted in the seventh live broadcast screen, so as to achieve the display effect of highlighting the preset objects in the live broadcast screen, so as to attract audience users to focus on the preset objects in the live broadcast screen, thereby improving the live broadcast effect.

[0192] It should be noted that, based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0193] based on Figure 2 Corresponding to the adjustment method based on the live screen provided in the embodiment, the embodiment of the present application also provides an adjustment device based on the live screen, see Fig.12 , Fig.12 A structural diagram of an adjustment device based on a live picture provided in an embodiment of the present application, the adjustment device based on a live picture 1200 comprises: a detection unit 1201, a segmentation unit 1202, an adjustment unit 1203 and a fusion unit 1204;

[0194] The detection unit 1201 is used to perform object detection on the screen content of the first live screen, and obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen;

[0195] The segmentation unit 1202 is used to segment the target object in the target detection frame from the first live broadcast picture according to the target detection frames in the multiple candidate detection frames, and obtain the second live broadcast picture and the first object image of the target object;

[0196] An adjustment unit 1203 is used to adjust the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of a plurality of preset objects;

[0197] The fusion unit 1204 is used to perform image fusion on the second object image and the second live picture to obtain a third live picture.

[0198] In a possible implementation manner, the preset adjustment strategy includes one or more of a size adjustment strategy and a position adjustment strategy.

[0199] In a possible implementation, when the preset adjustment strategy is a size adjustment strategy, the size adjustment strategy is a size upsampling strategy, and the adjustment unit 1203 is specifically configured to:

[0200] Enlarging the first object image according to a size upsampling strategy to obtain a second object image; or,

[0201] The size adjustment strategy is a size downsampling strategy, and the adjustment unit 1203 is specifically used to:

[0202] The first object image is downsized according to a size downsampling strategy to obtain a second object image.

[0203] In a possible implementation, when the preset adjustment strategy is a size adjustment strategy, the size adjustment strategy is a size enlargement model, and the adjustment unit 1203 is specifically configured to:

[0204] The first object image is enlarged according to the size enlargement model to obtain the second object image; the size enlargement model is used to enlarge the size of the first object image according to the mapping relationship between the first sample image and the second sample image, and the resolution of the first sample image is smaller than the resolution of the second sample image; or,

[0205] The size adjustment strategy is a size reduction model, and the adjustment unit 1203 is specifically used to:

[0206] The first object image is reduced in size according to the size reduction model to obtain a second object image; the size reduction model is used to reduce the size of the first object image according to a mapping relationship between the third sample image and the fourth sample image, and the resolution of the third sample image is greater than that of the fourth sample image.

[0207] In a possible implementation manner, when the preset adjustment strategy is a position adjustment strategy, the adjustment unit 1203 is specifically configured to:

[0208] The first object image is moved according to the position adjustment strategy to obtain a second object image; the position of the second object image is different from the position of the first object image.

[0209] In a possible implementation, the fusion unit 1204 is specifically configured to:

[0210] If it is determined according to the second object image and the second live picture that the second live picture includes the area to be filled, a similar area around the area to be filled in the second live picture is determined as a reference filling area;

[0211] Filling the area to be filled in the second live picture with content according to the area content of the reference filling area, to obtain a fourth live picture;

[0212] The second object image and the fourth live picture are image-fused to obtain a third live picture.

[0213] In a possible implementation, the device further includes: a smoothing unit;

[0214] a smoothing unit, configured to smooth the filled area to be filled in the fourth live picture according to the second live picture, so as to obtain a fifth live picture;

[0215] The fusion unit 1204 is specifically configured to:

[0216] The second object image and the fifth live picture are image-fused to obtain a third live picture.

[0217] In a possible implementation, the smoothing unit is further configured to:

[0218] According to the second live picture in the third live picture, a smoothing process is performed on the second object image in the third live picture to obtain a sixth live picture.

[0219] In a possible implementation, the smoothing unit is specifically configured to:

[0220] Acquire a first gradient field of a second object image in a third live picture, and a second gradient field of a second live picture in the third live picture;

[0221] The second object image in the third live picture is smoothed according to the first gradient field and the second gradient field to obtain a sixth live picture.

[0222] In a possible implementation, the smoothing unit is specifically configured to:

[0223] The second object image in the third live picture is smoothed according to a mapping relationship between the second object image and the second live picture by using an image smoothing model to obtain a sixth live picture.

[0224] In a possible implementation, the detection unit 1201 is specifically configured to:

[0225] Object detection is performed on the picture content of the first live picture according to the target frequency to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture.

[0226] In a possible implementation, the device further includes: an updating unit;

[0227] Update unit, specifically used for:

[0228] If the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the ith frame, the target frequency is updated according to the maximum frequency, where i is a positive integer;

[0229] If the multiple preset objects in the first live broadcast picture of the i+1th frame do not move relative to the multiple preset objects in the first live broadcast picture of the ith frame, the target frequency is updated according to the difference frequency between the target frequency and the preset frequency; the updated target frequency is greater than or equal to the minimum frequency.

[0230] In a possible implementation manner, the apparatus further includes: a determining unit;

[0231] Determine the unit, specifically for:

[0232] Acquire multiple coordinate differences between multiple pixel coordinates of the first object in the first live picture of the (i+1)th frame and multiple pixel coordinates of the first object in the first live picture of the (i)th frame;

[0233] If a preset number of coordinate differences among the plurality of coordinate differences is greater than a preset difference, it is determined that the first object in the (i+1)th frame of the first live picture moves relative to the first object in the (i)th frame of the first live picture.

[0234] In a possible implementation, the preset adjustment strategy further includes a replacement adjustment strategy; the adjustment unit 1203 is specifically configured to:

[0235] The first object image is replaced according to a preset replacement image through a replacement adjustment strategy to obtain a second object image.

[0236] It can be seen from the above technical scheme that, for the screen content of the first live screen, object detection obtains multiple candidate detection frames corresponding to multiple preset objects in the first live screen; for the target detection frames in the multiple candidate detection frames, the target object in the target detection frame is segmented from the first live screen to obtain the second live screen and the first object image of the target object; through the detection unit and the segmentation unit, the first live screen can be accurately segmented into the first object image of the target object to be adjusted and the second live screen to be merged. According to the preset adjustment strategy for highlighting one or more of the multiple preset objects, the first object image is adjusted to obtain the second object image; the second object image and the second live screen are merged to obtain the third live screen; through the adjustment unit and the fusion unit based on the preset adjustment strategy, the first object image of the target object is changed into the second object image, and merged into the second live screen to obtain the third live screen, so that the third live screen can highlight one or more of the multiple preset objects. Based on this, the method can achieve the display effect of highlighting the preset object in the live screen by detecting, segmenting, adjusting the target object in the live screen and re-merging it into the live screen without changing the size of the live screen, so as to attract the audience to focus on the preset object in the live screen, thereby improving the live effect.

[0237] The present application also provides a computer device, which may be a terminal. Fig.13 , Fig.13A structural diagram of a terminal provided for an embodiment of the present application. Taking a smartphone as an example, the smartphone includes: a radio frequency (RF) circuit 1310, a memory 1320, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a wireless fidelity (WiFi) module 1370, a processor 1380, and a power supply 13120 and other components. The input unit 1330 may include a touch panel 1331 and other input devices 1332, the display unit 1340 may include a display panel 1341, and the audio circuit 1360 may include a speaker 1361 and a microphone 1362. Those skilled in the art will appreciate that Fig.13 The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0238] The memory 1320 can be used to store software programs and modules. The processor 1380 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1320. The memory 1320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 1320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0239] The processor 1380 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in the memory 1320, and calling data stored in the memory 1320. Optionally, the processor 1380 may include one or more processing units; preferably, the processor 1380 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1380.

[0240] In this embodiment, the processor 1380 in the smart phone can execute the methods provided in various optional implementations of the above embodiments.

[0241] The computer device provided in the embodiment of the present application may also be a server. Fig.14 , Fig.14 A structural diagram of a server provided in an embodiment of the present application, the server 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors, such as CPU 1422, and memory 1432, one or more storage media 1430 (such as one or more mass storage devices) storing application programs 1442 or data 1444. Among them, the memory 1432 and the storage medium 1430 can be short-term storage or persistent storage. The program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1422 can be configured to communicate with the storage medium 1430 and execute a series of instruction operations in the storage medium 1430 on the server 1400.

[0242] The server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0243] In this embodiment, the central processor 1422 in the server 1400 can execute the methods provided in various optional implementations of the above embodiments.

[0244] According to one aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store a computer program. When the computer program is executed on a computer device, the computer device executes the methods provided in various optional implementations of the above embodiments.

[0245] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional implementations of the above-mentioned embodiments.

[0246] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0247] The terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0248] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0249] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0250] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0251] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), RAM, disk or optical disk and other media that can store computer programs.

[0252] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technical members in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for adjusting a live broadcast screen, characterized in that: The method comprises: Performing object detection on the screen content of the first live screen to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live screen; According to the target detection frames in the multiple candidate detection frames, segmenting the target object in the target detection frame from the first live picture, and obtaining a second live picture and a first object image of the target object; Adjust the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of the multiple preset objects; The second object image and the second live broadcast picture are fused to obtain a third live broadcast picture.

2. The method according to claim 1, characterized in that The preset adjustment strategy includes one or more of a size adjustment strategy and a position adjustment strategy.

3. The method according to claim 2, characterized in that When the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is a size upsampling strategy, and the first object image is adjusted according to the preset adjustment strategy to obtain the second object image, specifically: The first object image is enlarged according to the size upsampling strategy to obtain the second object image; or, The size adjustment strategy is a size downsampling strategy, and the first object image is adjusted according to the preset adjustment strategy to obtain the second object image, specifically: The first object image is downsized according to the size downsampling strategy to obtain the second object image.

4. The method according to claim 2, characterized in that: When the preset adjustment strategy is the size adjustment strategy, the size adjustment strategy is a size magnification model, the first object image is adjusted according to the preset adjustment strategy to obtain the second object image, specifically: The first object image is enlarged according to the size enlargement model to obtain the second object image; the size enlargement model is used to enlarge the size of the first object image according to the mapping relationship between the first sample image and the second sample image, and the resolution of the first sample image is smaller than the resolution of the second sample image; or, The size adjustment strategy is a size reduction model, and the first object image is adjusted according to the preset adjustment strategy to obtain the second object image, specifically: The first object image is reduced in size according to the size reduction model to obtain the second object image; the size reduction model is used to reduce the size of the first object image according to a mapping relationship between a third sample image and a fourth sample image, and a resolution of the third sample image is greater than a resolution of the fourth sample image.

5. The method according to claim 2, characterized in that: When the preset adjustment strategy is the position adjustment strategy, the first object image is adjusted according to the preset adjustment strategy to obtain the second object image, specifically: The first object image is moved according to the position adjustment strategy to obtain the second object image; the position of the second object image is different from the position of the first object image.

6. The method according to claim 1, characterized in that The performing image fusion on the second object image and the second live broadcast picture to obtain a third live broadcast picture includes: If it is determined according to the second object image and the second live picture that the second live picture includes a to-be-filled area, a similar area around the to-be-filled area in the second live picture is determined as a reference filling area; Filling the to-be-filled area in the second live picture with content according to the area content of the reference filling area to obtain a fourth live picture; The second object image and the fourth live broadcast picture are fused to obtain the third live broadcast picture.

7. The method according to claim 6, characterized in that The method further comprises: According to the second live broadcast picture, a smoothing process is performed on the filled area to be filled in the fourth live broadcast picture to obtain a fifth live broadcast picture; The performing image fusion on the second object image and the fourth live picture to obtain the third live picture is specifically: The second object image and the fifth live broadcast picture are fused to obtain the third live broadcast picture.

8. The method according to any one of claims 1 to 7, characterized in that: After obtaining the third live picture, the method further includes: According to the second live picture in the third live picture, the second object image in the third live picture is smoothed to obtain a sixth live picture.

9. The method according to claim 8, characterized in that The step of performing smoothing processing on the second object image in the third live picture according to the second live picture in the third live picture to obtain a sixth live picture includes: Acquire a first gradient field of the second object image in the third live picture, and a second gradient field of the second live picture in the third live picture; The second object image in the third live picture is smoothed according to the first gradient field and the second gradient field to obtain the sixth live picture.

10. The method according to claim 8, characterized in that The step of performing smoothing processing on the second object image in the third live picture according to the second live picture in the third live picture to obtain a sixth live picture is specifically as follows: The second object image in the third live picture is smoothed by an image smoothing model according to a mapping relationship between the second object image and the second live picture to obtain the sixth live picture.

11. The method according to claim 1, characterized in that: The performing object detection on the picture content of the first live picture to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture is specifically: Object detection is performed on the picture content of the first live picture according to the target frequency to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture.

12. The method according to claim 11, characterized in that The step of obtaining the target frequency comprises: If the first object in the first live broadcast picture of the i+1th frame moves relative to the first object in the first live broadcast picture of the ith frame, the target frequency is updated according to the maximum frequency, where i is a positive integer; If the multiple preset objects in the first live broadcast picture of the i+1th frame do not move relative to the multiple preset objects in the first live broadcast picture of the i-th frame, the target frequency is updated according to the difference frequency between the target frequency and the preset frequency; the updated target frequency is greater than or equal to the minimum frequency.

13. The method according to claim 12, characterized in that The step of determining whether the first object in the first live picture of the (i+1)th frame moves relative to the first object in the first live picture of the i-th frame includes: Acquire multiple coordinate differences between multiple pixel coordinates of the first object in the (i+1)th frame of the first live picture and multiple pixel coordinates of the first object in the (i)th frame of the first live picture; If a preset number of coordinate differences among the multiple coordinate differences is greater than a preset difference, it is determined that the first object in the (i+1)th frame of the first live picture moves relative to the first object in the (i)th frame of the first live picture.

14. The method according to claim 2, characterized in that The preset adjustment strategy also includes a replacement adjustment strategy; the first object image is adjusted according to the preset adjustment strategy to obtain the second object image, specifically: The first object image is replaced according to a preset replacement image using the replacement adjustment strategy to obtain the second object image.

15. An adjustment device based on live broadcast picture, characterized in that: The device comprises: a detection unit, a segmentation unit, an adjustment unit and a fusion unit; The detection unit is used to perform object detection on the picture content of the first live picture to obtain multiple candidate detection frames corresponding to multiple preset objects in the first live picture; The segmentation unit is used to segment the target object in the target detection frame from the first live picture according to the target detection frames in the multiple candidate detection frames, and obtain a second live picture and a first object image of the target object; The adjustment unit is used to adjust the first object image according to a preset adjustment strategy to obtain a second object image; the preset adjustment strategy is used to highlight one or more of the multiple preset objects; The fusion unit is used to perform image fusion on the second object image and the second live broadcast picture to obtain a third live broadcast picture.

16. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 14 according to the instructions in the computer program.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed on a computer device, the computer device is enabled to execute the method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 14.

Images