Video processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video processing method and device, electronic equipment and a storage medium, relates to the technical field of video processing, and aims to see a leading portrait more easily and clearly. The video processing method comprises the steps that in a first time period, a first video picture is displayed, in the first time period, the distance between a first user and a camera is smaller than the distance between a second user and the camera, the distance between the first user and the camera is smaller than the distance between a third user and the camera, and the first video picture comprises a portrait of the first user; after the first user moves away from the camera, a second video picture is displayed, and the second video picture comprises portraits of a second user and a third user; after the fourth user moves into the capturing range of the camera, a third video picture is displayed, and the third video picture comprises portraits of the first user, the second user, the third user and the fourth user.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] With the development of technology, video shooting is applied in more and more scenarios. In some scenarios, a single video screen cannot meet the needs. For example, when making a video call through a TV, due to the wide angle of the camera, the camera screen is large, while the person in the screen is small, resulting in the other party not being able to see the person on the video call clearly. Summary of the invention

[0003] The technical solution of the present application provides a video processing method, device, electronic device and storage medium, which can make it easier to see the portrait of the protagonist.

[0004] In a first aspect, a video processing method is provided, comprising: in a first time period, displaying a first video screen, wherein in the first time period, a distance between a first user and a camera is smaller than a distance between a second user and the camera, and a distance between the first user and the camera is smaller than a distance between a third user and the camera, and the first video screen includes a portrait of the first user; after the first user moves away from the camera, displaying a second video screen, and the second video screen includes portraits of the second user and the third user; after a fourth user moves into a capture range of the camera, displaying a third video screen, wherein a distance between the fourth user and the first user is smaller than a distance between the second user and the first user, and a distance between the fourth user and the second user is larger than a distance between the second user and the third user, and the third video screen includes portraits of the first user, the second user, the third user, and the fourth user.

[0005] In a possible implementation, in a first time period, the first user is located in a third area, and the second user and the third user are located in a fourth area; in the first time period, displaying the first video image includes: based on the fact that in the first time period, the distance between the first user and the camera is smaller than the distance between the second user and the camera, and the distance between the first user and the camera is smaller than the distance between the third user and the camera, displaying the first video image.

[0006] In a possible implementation, after the first user moves away from the camera, displaying the second video screen includes: after the first user moves away from the camera, in a second time period, based on the fact that within the second time period, the distance between the first user and the camera is equal to the distance between the second user and the camera, the distance between the first user and the camera is equal to the distance between the third user and the camera, and the number of users in the fourth area is greater than the number of users in the third area, the second video screen is displayed.

[0007] In a possible implementation, during the process from the first time period to the second time period, the video display interface gradually changes from displaying the first video picture to displaying the second video picture.

[0008] In one possible implementation, in the first time period, displaying the first video image includes: obtaining a second target area including at least one portrait in the original video image captured by the camera in the first time period; obtaining a third target area of ​​the first time period based on the second target area of ​​the first time period and the third target area of ​​the previous time period adjacent to the first time period, wherein the coordinates of the third target area of ​​the first time period are closer to the coordinates of the second target area of ​​the first time period relative to the coordinates of the third target area of ​​the previous time period adjacent to the first time period, or the coordinates of the third target area of ​​the first time period are equal to the coordinates of the second target area of ​​the first time period; based on the third target area of ​​the first time period, the original video image captured by the camera in the first time period is cropped to obtain a cropped first video image, and the first video image is displayed.

[0009] In a possible implementation, the video processing method further includes: determining a target portrait from portraits in an original video image captured by a camera during a first period of time, obtaining a first target area for the first period of time, the first target area for the first period of time being a minimum rectangular frame including all target portraits; obtaining a second target area including at least one portrait from the original video image captured by the camera during the first period of time includes: if a non-calculation condition is met, taking a second target area of ​​a previous period of time adjacent to the first period of time as a second target area for the first period of time; if the non-calculation condition is not met, obtaining a second target area including the first target area of ​​the first period of time in the original video image of the first period of time, and the second target area including the first target area of ​​the first period of time is obtained from ... The size of the second target area in one time period is larger than the size of the first target area in the first time period; the non-counting condition includes at least one of the following items or any combination thereof. When the non-counting condition includes multiple items, the non-counting condition is not satisfied if any one of the multiple items is not satisfied, and the non-counting condition is satisfied if all of the multiple items are satisfied: the preset resolution level to which the first target area in the previous time period adjacent to the first time period belongs is the same as the preset resolution level to which the first target area in the first time period belongs; the first target area in the first time period is covered by the second target area in the previous time period adjacent to the first time period; the number of target portraits in the first time period is equal to the number of target portraits in the previous time period adjacent to the first time period.

[0010] In a possible implementation, in a first time period, the first user is located in a third area, and the second user and the third user are located in a fourth area; the portrait corresponding to the user whose camera distance in the original video screen is closest is the first portrait, and the other portraits are the second portraits, and the first portrait and all the second portraits whose camera distance difference with the first portrait is ≤ the preset distance are portraits to be determined, and the original video screen includes a first area corresponding to the third area and a second area corresponding to the fourth area; determining the target portrait from the portraits in the original video screen captured by the camera in the first time period includes: if the number of portraits to be determined in the first area and the second area of ​​the original video screen captured by the camera in the first time period is not equal, then all the portraits to be determined in the first area and the second area of ​​the original video screen captured by the camera in the first time period, whichever has the larger number of portraits to be determined, are determined as target portraits; if the number of portraits to be determined in the first area and the second area of ​​the original video screen captured by the camera in the first time period is equal, then all the portraits to be determined in the original video screen captured by the camera in the first time period are determined as target portraits.

[0011] In a possible implementation, based on the second target area of ​​the first time period and the third target area of ​​the previous time period adjacent to the first time period, the process of acquiring the third target area of ​​the first time period includes: if the second target area of ​​the first time period is reduced relative to the third target area of ​​the previous time period adjacent to the first time period, the third target area of ​​the previous time period adjacent to the first time period is reduced to obtain the third target area of ​​the first time period; if the second target area of ​​the first time period is the same size as the third target area of ​​the previous time period adjacent to the first time period, and the second target area of ​​the first time period is different from the third target area of ​​the previous time period adjacent to the first time period in terms of central coordinate position, the third target area of ​​the previous time period adjacent to the first time period is moved in a direction close to the second target area of ​​the first time period to obtain the third target area of ​​the first time period; if the second target area of ​​the first time period is enlarged relative to the third target area of ​​the previous time period adjacent to the first time period, and the second target area of ​​the first time period is the same size as the third target area of ​​the previous time period adjacent to the first time period in terms of central coordinate position, the third target area of ​​the previous time period adjacent to the first time period is enlarged to obtain the third target area of ​​the first time period.

[0012] In a possible implementation, the process of reducing the third target area of ​​the previous time period adjacent to the first time period to obtain the third target area of ​​the first time period includes: reducing 10 times the coordinate value of the third target area of ​​the previous time period adjacent to the first time period based on 10 times the step value by an integer int to obtain 10 times the coordinate value of the third target area of ​​the first time period; moving the third target area of ​​the previous time period adjacent to the first time period in a direction close to the second target area of ​​the first time period to obtain the third target area of ​​the first time period includes: moving 10 times the coordinate value of the third target area of ​​the previous time period adjacent to the first time period based on 10 times the step value by an integer int to obtain the coordinate value of the third target area of ​​the first time period 10 times the coordinate value of the third target area of ​​the first time period; amplifying the third target area of ​​the previous time period adjacent to the first time period to obtain the third target area of ​​the first time period includes: amplifying 10 times the coordinate value of the third target area of ​​the previous time period adjacent to the first time period based on 10 times the step value through an integer int to obtain 10 times the coordinate value of the third target area of ​​the first time period; the process of cropping the original video picture of the first time period based on the third target area of ​​the first time period to obtain a cropped picture includes: dividing 10 times the coordinate value of the third target area of ​​the first time period by 10 to obtain the coordinate value of the third target area of ​​the first time period, and cropping the original video picture of the first time period based on the coordinate value of the third target area of ​​the first time period through a floating point method to obtain a cropped picture.

[0013] In a possible implementation, the video processing method further includes: obtaining the distance between the user and the camera corresponding to each portrait in the original video screen of the current time period captured by the camera; obtaining the number of portraits in the first area and the number of portraits in the second area in the original video screen of the current time period; displaying any one of the first video screen, displaying the second video screen, and displaying the third video screen includes: displaying the video screen based on the distance between each user and the camera, the number of portraits in the first area, and the number of portraits in the second area.

[0014] In a possible implementation, the video processing method further includes: obtaining a second target area including at least one portrait in the original video picture of the current time period; obtaining the third target area of ​​the current time period based on the second target area of ​​the current time period and the third target area of ​​the previous time period, wherein the coordinates of the third target area of ​​the current time period are closer to the coordinates of the second target area of ​​the current time period relative to the coordinates of the third target area of ​​the previous time period, or the coordinates of the third target area of ​​the current time period are equal to the coordinates of the second target area of ​​the current time period; based on the third target area of ​​the current time period, the original video picture of the current time period is cropped to obtain a cropped picture, and any one of the first video picture, the second video picture and the third video picture is the cropped picture; in multiple consecutive time periods, if the coordinates of the second target area remain unchanged, the coordinates of the third target area gradually approach the second target area.

[0015] In a possible implementation, the video processing method further includes: determining a target portrait among the portraits in the original video screen of the current time period, and obtaining a first target area of ​​the current time period, wherein the first target area of ​​the current time period is a minimum rectangular frame including all target portraits; obtaining a second target area including at least one portrait in the original video screen of the current time period includes: if a non-computation condition is met, taking the second target area of ​​the previous time period as the second target area of ​​the current time period; if the non-computation condition is not met, obtaining a second target area including the first target area of ​​the current time period in the original video screen of the current time period, wherein the size of the second target area of ​​the current time period is greater than the size of the first target area of ​​the current time period; the non-computation condition includes at least one of the following items or any combination thereof, and when the non-computation condition includes multiple items, if any one of the multiple items is not met, the non-computation condition is not met, and if all of the multiple items are met, the non-computation condition is met: the preset resolution level to which the first target area of ​​the previous time period belongs is the same as the preset resolution level to which the first target area of ​​the current time period belongs; the first target area of ​​the current time period is covered by the second target area of ​​the previous time period; the number of target portraits in the current time period is equal to the number of target portraits in the previous time period.

[0016] In a possible implementation, the portrait corresponding to the user whose camera distance in the original video screen is closest is the first portrait, the other portraits are the second portraits, and the first portrait and all the second portraits whose camera distance difference with the first portrait is ≤ the preset distance are the portraits to be determined; determining the target portrait among the portraits in the original video screen of the current time period includes: if the number of portraits to be determined in the first area and the second area of ​​the original video screen of the current time period is not equal, then all the portraits to be determined in the first area and the second area of ​​the original video screen of the current time period with the larger number of portraits to be determined are determined as target portraits; if the number of portraits to be determined in the first area and the second area of ​​the original video screen of the current time period is equal, then all the portraits to be determined in the original video screen of the current time period are determined as target portraits.

[0017] In a possible implementation, based on the second target area of ​​the current time period and the third target area of ​​the previous time period, the process of obtaining the third target area of ​​the current time period includes: if the second target area of ​​the current time period is reduced relative to the third target area of ​​the previous time period, the third target area of ​​the previous time period is reduced to obtain the third target area of ​​the current time period; if the second target area of ​​the current time period is the same size as the third target area of ​​the previous time period, and the center coordinate position of the second target area of ​​the current time period is different from that of the third target area of ​​the previous time period, then the third target area of ​​the previous time period is moved in a direction close to the second target area of ​​the current time period to obtain the third target area of ​​the current time period; if the second target area of ​​the current time period is enlarged relative to the third target area of ​​the previous time period, and the center coordinate position of the second target area of ​​the current time period is the same as that of the third target area of ​​the previous time period, then the third target area of ​​the previous time period is enlarged to obtain the third target area of ​​the current time period.

[0018] In a possible implementation, the video processing method further includes: reducing the third target area of ​​the previous time period, and the process of obtaining the third target area of ​​the current time period includes: reducing 10 times the coordinate value of the third target area of ​​the previous time period based on 10 times the step value by an integer int, and obtaining 10 times the coordinate value of the third target area of ​​the current time period; moving the third target area of ​​the previous time period toward the direction close to the second target area of ​​the current time period, and the process of obtaining the third target area of ​​the current time period includes: moving 10 times the coordinate value of the third target area of ​​the previous time period based on 10 times the step value by an integer int to the direction close to 10 times the coordinate value of the second target area of ​​the current time period, and obtaining the third target area of ​​the current time period. 10 times the coordinate value of the target area; the process of enlarging the third target area of ​​the previous time period to obtain the third target area of ​​the current time period includes: enlarging 10 times the coordinate value of the third target area of ​​the previous time period based on 10 times the step value through an integer int to obtain 10 times the coordinate value of the third target area of ​​the current time period; the process of cropping the original video picture of the current time period based on the third target area of ​​the current time period to obtain a cropped picture includes: dividing 10 times the coordinate value of the third target area of ​​the current time period by 10 to obtain the coordinate value of the third target area of ​​the current time period, and cropping the original video picture of the current time period based on the coordinate value of the third target area of ​​the current time period through a floating point method to obtain a cropped picture.

[0019] In a possible implementation manner, the aspect ratio of the third target area is the same as the aspect ratio of the original video picture.

[0020] In a possible implementation, displaying the first video screen includes displaying the first video screen in at least one or any combination of a video incoming call notification interface, a video phone dialing interface, and a video call interface.

[0021] In a second aspect, a video processing device is provided, including: a first display module, used to display a first video screen in a first time period, in which the distance between the first user and the camera is smaller than the distance between the second user and the camera, and the distance between the first user and the camera is smaller than the distance between the third user and the camera, and the first video screen includes a portrait of the first user; a second display module, used to display a second video screen after the first user moves away from the camera, and the second video screen includes portraits of the second user and the third user; a third display module, used to display a third video screen after the fourth user moves into the capture range of the camera, the distance between the fourth user and the first user is smaller than the distance between the second user and the first user, and the distance between the fourth user and the second user is larger than the distance between the second user and the third user, and the third video screen includes portraits of the first user, the second user, the third user and the fourth user.

[0022] In a third aspect, an electronic device is provided, including: a processor and a memory, the memory being used to store at least one instruction, and when the instruction is loaded and executed by the processor, the electronic device executes the above-mentioned video processing method.

[0023] In a fourth aspect, a computer-readable storage medium is provided, comprising a program or an instruction. When the program or the instruction is run on a computer, the above method is executed.

[0024] The video processing method, device, electronic device and storage medium in the embodiments of the present application redefine the protagonist based on the number of people corresponding to the portrait in the video picture and the change in the camera distance, and track and enlarge the newly determined protagonist through the cropped picture, thereby highlighting the protagonist portrait in the video picture, achieving focus on the protagonist, and making it easier to see the protagonist portrait. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a structural block diagram of an electronic device in an embodiment of the present application;

[0026] Figure 2 This is a software structure diagram in an embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of an interaction process between different layers in an embodiment of the present application;

[0028] Figure 4 This is a schematic diagram of an original video screen in an embodiment of the present application;

[0029] Figure 5 for Figure 3 Detailed flowchart of some steps in the figure;

[0030] Figure 6 for Figure 3 A specific flow chart of another part of the steps in FIG.

[0031] Figure 7 for Figure 6 Detailed flowchart of some steps in the figure;

[0032] Figure 8 for Figure 6 A specific flow chart of another part of the steps in the embodiment;

[0033] Fig. 9 for Figure 8 Detailed flowchart of some steps in the figure;

[0034] Fig.10 A schematic diagram of a relationship between a first target area, a second target area and an original video picture in an embodiment of the present application;

[0035] Fig.11 for Figure 6 A specific flow chart of another part of the steps in the embodiment;

[0036] Fig.12 This is a schematic diagram of a third target area gradually changing in a plurality of consecutive time periods in an embodiment of the present application;

[0037] Fig.13 This is a schematic diagram of a user interface change in an embodiment of the present application;

[0038] Fig.14 A schematic diagram of an original video screen in an embodiment of the present application;

[0039] Fig.15 A schematic diagram of another original video screen in an embodiment of the present application;

[0040] Fig.16 A schematic diagram of another original video screen in an embodiment of the present application;

[0041] Fig.17 This is a schematic diagram of another original video screen in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0043] Figure 1 A schematic structural diagram of an electronic device 100 is shown.

[0044] The electronic device 100 may include a processor 110 , an internal memory 121 , an audio module 170 , a speaker 170A, a microphone 170C, a camera 193 , a display screen 194 , and the like.

[0045] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0046] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0047] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0048] The communication function of the electronic device 100 can be implemented in a wired or wireless manner.

[0049] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0050] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0051] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0052] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0053] The internal memory 121 may be used to store computer executable program codes, wherein the executable program codes include instructions.

[0054] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0055] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or make calls through the speaker 170A.

[0056] The microphone 170C, also called a "microphone" or a "speaker", is used to convert a sound signal into an electrical signal. When making a call, the user can input a sound signal into the microphone 170C.

[0057] The following takes a smart TV as an example of an electronic device to illustrate a process of applying an embodiment of the present application in a video call scenario of the smart TV.

[0058] First, the embodiment of the present application is described in conjunction with the software architecture, and the software structure of the electronic device 100 is exemplified. Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0059] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software system includes an application layer, an application framework layer, and a hardware abstraction layer (HAL) from top to bottom. The application layer may include applications such as cameras and video call applications. The application framework layer may include a camera service module CameraSevice, etc. The hardware abstraction layer is used to process the video stream captured by the camera. The hardware abstraction layer may include an image acquisition module, an algorithm detection module, and an image cropping module. Each of the above modules may be a process or a thread.

[0060] like Figure 2 and Figure 3As shown, the embodiment of the present application provides a video processing method, which can be applied to the above-mentioned electronic device. The method may include the following steps. Step 101, the video call application sends a stream allocation request to the camera service module in response to a video call request or a video call notification request. The video call request is generated by the event of the user actively initiating a call, and the video call notification request is generated by the event of receiving the other party's call. The stream allocation request is used to request to open the camera and configure the camera's frame rate and resolution and other video stream parameters. In some embodiments, in response to the video call notification request, the stream allocation request is triggered to be sent to the camera service module to trigger the subsequent acquisition of the video stream captured by the camera to realize the video preview function; in other embodiments, the video call notification request in the above step 101 can be replaced by a video call connection request, that is, before the user connects the video call, it is not necessary to obtain the video stream captured by the camera, but to obtain it when the call is connected. In step 101, the video call application can specifically call the configstream (stream allocation) function to send a stream allocation request to the camera service module. Step 102, the camera service module sends the stream allocation information to the image acquisition module in the hardware abstraction layer in response to the configuration request. Step 103, the image acquisition module performs stream allocation in response to the stream allocation information. The stream allocation process includes configuring the frame rate and resolution of the camera. Step 104 is executed after step 103. The video call application sends a video stream acquisition request to the camera service module. The video call application can specifically call the RepeatRequest function to send a video stream acquisition request to the camera service module. Step 105. The camera service module sends a video stream acquisition request to the image acquisition module in response to the video stream acquisition request. Step 106. The image acquisition module sends a video stream acquisition request to the camera in response to the video stream acquisition request. Step 107. The image acquisition module acquires the original video screen, specifically, periodically acquires the original video screen acquired by the camera in time periods. For example, every 30 ms is a time period. The image acquisition module sends a video stream acquisition request to the camera every 30 ms, and acquires the original video screen of the current time period from the camera, that is, an image acquisition is performed. The acquired image will be decoded, and the decoded original video screen will be provided to the algorithm detection module, that is, step 108 is executed. The algorithm detection module acquires the original video screen, for example, Figure 4As shown, the original video picture of the current time period obtained from the camera is a 1920*1024 h264-encoded image, which is a 1920*1024 YUYV format image after decoding. Step 109, the algorithm detection module performs portrait detection based on the original video picture of the current time period, obtains portrait detection information, and sends the portrait detection information to the image cropping module. Step 110, the image cropping module crops the original video picture based on the portrait detection information. In each time period, the image cropping module obtains the third target area C of the current time period based on the portrait detection information of the current time period. The third target area C refers to the actual cropping frame of the current picture. Obtaining the third target area C means obtaining the coordinates of the left frame, right frame, upper frame and lower frame corresponding to the third target area C, and then crops the original video picture of the current time period based on the third target area C. Step 110, the camera service module obtains the cropped picture. Step 111, the video call application obtains the cropped picture. The screen finally displayed by the video call application is the screen after the original video screen is cropped based on the third target area of ​​the current time period. The size of the cropped screen is usually smaller than the size of the original video screen, so its size is adjusted to 1080P, and the cropped screen is transmitted to the video call application for display. That is, in each time period, the image cropping module crops the currently acquired original video screen based on the third target area of ​​the current time period and transmits it to the video call application to display the cropped screen. The above steps 107 to 111 can be performed periodically, that is, each time period will perform portrait detection based on the original video screen of the current time period, and crop based on the portrait detection information of the current time period.

[0061] The above steps 101 to 106 are described in detail below. Figure 5 As shown, the electronic device may include an application package 401 of a first application, a camera manager (for example, recorded as CameraManager) 402, a camera device instance (for example, recorded as CameraDeviceImpl) 403, a camera device client (for example, recorded as CameraDeviceClient) 404, a camera device (for example, recorded as Camera3Device) 405, and a camera device session (for example, recorded as CameraDeviceSession) 406.

[0062] In one embodiment, the first application may be a video call application corresponding to a video call scene. In another embodiment, the first application may be a camera application corresponding to a camera preview scene or a camera recording scene.

[0063] In one embodiment, the application package 401 may be an APK (Android application package) of the first application.

[0064] In one embodiment, Camera 2 API (Application Programming Interface) may include a camera manager 402 and a camera device instance 403 .

[0065] In one embodiment, the CameraService (camera service module) in the application framework layer of the electronic device may include a camera device client 404 and a camera device 405 .

[0066] In one embodiment, the CameraHal (camera hardware abstraction layer) of the electronic device may include a camera device session 406 .

[0067] In one embodiment of the present application, the camera device session 406 may include an image acquisition module, an algorithm detection module, and an image cropping module. For details about the implementation of the functions of these three modules, please refer to Figure 3 Description of related technologies in the illustrated embodiments.

[0068] like Figure 7 As shown, the process of steps 101 to 106 may include the following contents:

[0069] Step 4.1, in response to the first instruction, the application package 401 sends a request for obtaining a camera identification number (Identity document, id) to the camera manager 402, so that the camera manager 402 returns a list of camera identification numbers.

[0070] The camera manager 402 may provide the application package 401 with a camera identification number list according to the received camera identification number acquisition request, so that the application package 401 can acquire the camera identification number accordingly.

[0071] In one embodiment, the relevant implementation code for sending the request for obtaining the camera identification number to the camera manager 402 may include: CameraManager.getCameraIdList().

[0072] In one embodiment of the present application, when the first application is a video call application, the first instruction may be an instruction to start a video call. In one feasible implementation, the user may trigger a video initiation control in the video call application to issue an instruction to start a video call.

[0073] Step 4.2, the application package 401 obtains the camera identification number according to the camera identification number list sent by the camera manager 402. In one embodiment, the relevant implementation code for obtaining the camera ID may include: CameraManager.getCameraCharacteristics(CameraId).

[0074] In a feasible implementation, the acquired camera identification number may be the identification number of a camera used for image acquisition, and the image acquisition module in the camera device session 406 may acquire the image captured by the camera.

[0075] Step 4.3, the application package 401 determines whether the key of the corresponding camera exists according to the obtained camera identification number, and if so, executes step 4.4.

[0076] In a feasible implementation, if it is determined that the key of the corresponding camera does not exist, step 4.4 may not be performed.

[0077] In one embodiment, the relevant implementation code for determining whether a Key exists may include: mCharacteristics.getKeys().contains(HIHONOR_FACEAUTOTRACK_SUPPORTED)

[0078] KEY NAME: com.hihonor.device.capabilities.faceAutoTrackModeSupportedByte type.

[0079] In step 4.4, the application package 401 determines whether the corresponding application supports portrait tracking according to the obtained camera identification number, and if it does, it triggers the camera manager 402 to open the corresponding camera according to the camera key. The camera in the open state can sequentially capture each frame of the image within its image acquisition range.

[0080] In another embodiment of the present application, step 4.4 may be performed first and then step 4.3, that is, step 4.3 is performed when step 4.4 is judged to be passed, and step 4.5 is performed when step 4.3 is judged to be passed.

[0081] In one embodiment, the relevant implementation code for determining whether the corresponding application supports portrait tracking may include: mCharacteristics.get(HIHONOR_FACEAUTOTRACK_SUPPORTED) 0 Not supported 1 Supported byte type.

[0082] Since portrait detection can be performed on the images captured by the camera to achieve user portrait tracking, and then the image can be cropped according to the portrait detection information to display the cropped image, if the judgment result is "not supported", the camera manager 402 will not be triggered to open the corresponding camera. If the judgment result is "supported", the camera manager 402 will be triggered to open the corresponding camera.

[0083] In one embodiment, the relevant implementation code for triggering the camera manager 402 to open the corresponding camera may include: CameraManager.openCamera().

[0084] In step 4.5, the application package 401 sends an instruction for creating a capture request to the camera device instance 403. In one embodiment, the relevant implementation code of step 4.5 may include: CameraDevice.createCaptureRequest().

[0085] In step 4.6, the camera device instance 403 sends an instruction for creating a system setting request to the camera device client 404 according to the instruction for creating a capture request. In one embodiment, the relevant implementation code of step 4.6 may include: createDefaultRequest(templateType).

[0086] In step 4.7, the camera device client 404 transmits the received instruction of creating a system setting request to the camera device 405.

[0087] In step 4.8, the camera device 405 sends the instruction for setting the system setting request to the camera device session 406 according to the instruction for creating the system setting request. In one embodiment, the relevant implementation code of step 4.8 may include: constructDefaultRequestSettings.

[0088] Step 4.9, the camera device session 406 obtains the camera metadata and the current application package name according to the instruction of setting the system setting request, and returns the obtained camera metadata and the current application package name to the camera device 405.

[0089] In one embodiment, the relevant implementation code of step 4.9 may include: com.hihonor.capture.metadata.packageName.

[0090] In one embodiment, the camera metadata acquired by the camera device session 406 may be metadata of an opened camera.

[0091] In one embodiment, the current application package name may be the package name of the application program package 401 of the first application.

[0092] In step 4.10, the camera device 405 adds the current application package name to the camera metadata, and sends the camera metadata with the current application package name added to the camera device client 404. By adding the current application package name to the camera metadata, the scene information of the current period is brought in.

[0093] In step 4.11, the camera device client 404 transmits the camera metadata with the current application package name added to it to the application package 401.

[0094] By passing the camera metadata with the current application package name added thereto to the application package 401 , the capture request builder (CaptureRequest.Builder) of the application package 401 can include the package name and the device capability of the camera.

[0095] After the application package 401 obtains the camera metadata with scene information, it can start the stream distribution process (see steps 4.12 to 4.16).

[0096] Step 4.12: the application package 401 generates an instruction for creating a capture session according to the camera metadata to which the current application package name is added, and sends the instruction for creating a capture session to the camera device instance 403.

[0097] In one embodiment, the relevant implementation code of step 4.12 may include: CameraDevice.CreateCaptureSession(SessionConfiguration config).

[0098] Step 4.13, the camera device instance 403 sends the interface flow distribution information and the information of configuring the session key (sessionkey) to the camera device client 404 according to the instruction of creating the capture session.

[0099] In one embodiment, the flow allocation information may include frame rate and resolution. The flow allocation module of the Hal layer may configure the flow according to the flow allocation information, and configure the frame rate and resolution for the camera.

[0100] Step 4.14, the camera device client 404 processes the received interface flow distribution information and the information for configuring the session key, and sends the processed session key and the flow distribution information corresponding to the camera hardware abstraction layer (camera hal layer) to the camera device 405.

[0101] Step 4.15, the camera device 405 sends the session key and the stream allocation information corresponding to the camera hardware abstraction layer to the camera device session 406.

[0102] Step 4.16, the camera device session 406 performs stream allocation processing according to the session key and the stream allocation information of the corresponding camera hardware abstraction layer, and sends a stream allocation completion notification to the application package 401 after the stream allocation processing is completed.

[0103] In one embodiment, the flow distribution process may include mode selection and pipeline (transmission) configuration.

[0104] In one embodiment, after the flow distribution process is completed, the information “a” may be sent to the application package 401, so that the application package 401 knows that the flow distribution process is completed.

[0105] In one embodiment, the technical implementation of the flow distribution processing can refer to Figure 3 The relevant technical contents in the illustrated embodiment will not be elaborated in detail in this embodiment.

[0106] After the camera device session 406 completes the streaming process, the application package 401 may send a repeat request to trigger the camera device session 406 to perform image processing (such as algorithm detection and cropping processing) on ​​the image captured by the camera through the camera service module.

[0107] In step 4.17, the application package 401 sends an instruction for setting a repeat request to the camera device instance 403. In one embodiment, the relevant implementation code of step 4.17 may include: captureSession.setRepeatingRequest.

[0108] Step 4.18, the camera device instance 403 sends an instruction to submit a request list (for example, recorded as submitRequesList) to the camera device client 404 according to the instruction to set a repeated request.

[0109] Step 4.19, the camera device client 404 sends an instruction for setting a streaming request list (for example, recorded as setStreamingRequestList) to the camera device 405 according to the instruction for submitting the request list.

[0110] Step 4.20: the camera device 405 sends an image acquisition request to the camera device session 406 according to the instruction for setting the streaming request list.

[0111] Step 4.21, the camera device session 406 acquires a frame of image captured by the camera according to the image acquisition request, so as to subsequently process the acquired image to obtain a processed image, and send the processed image to the application package 401 for display.

[0112] In one embodiment of the present application, the camera device session 406 may include an image acquisition module, an algorithm detection module, and an image cropping module. The image acquisition module acquires images from the turned-on camera, and the acquired images are placed in an image buffer queue, and the algorithm detection module and the image cropping module asynchronously acquire images from the image buffer queue.

[0113] The algorithm detection module performs portrait detection on the acquired image, and the generated portrait detection information is stored in the portrait detection information buffer queue. After acquiring the image, the image cropping module attempts to obtain the portrait detection information from the portrait detection information buffer queue. If the portrait detection information is acquired, the image is cropped accordingly, and the cropped image is used as the capture result (for example, recorded as CaptureResult). If the portrait detection information is not acquired, no cropping is performed, and the acquired image is directly used as the capture result, or the image is cropped according to the portrait detection information used in the last cropping, and the cropped image is used as the capture result.

[0114] The image cropping module then transmits the captured result back to the application package 401 for display. In this way, the camera device session 406 completes the processing of the image acquisition request (for example, recorded as processcapturerequest). The specific implementation of image processing is described in detail in the subsequent content.

[0115] The specific process of step 109 and step 110 is described below. Before executing step 109, the original video picture of the current period has been obtained, such as Figure 4 As shown, the coordinate system is defined based on the boundary of the original video screen. The coordinates of any position in the coordinate system are (x, y), where x represents the horizontal coordinate and y represents the vertical coordinate. The upper left corner of the screen is the origin coordinate (0, 0), the upper right corner of the screen is (1920, 0), the lower left corner of the screen is (0, 1080), and the lower right corner of the screen is (1920, 1080). The coordinate unit can be pixel, that is, the size of the original video screen is 1920*1080 pixels, or the resolution of the original video screen is 1920*1080.

[0116] Step 109 may specifically be to perform portrait detection based on the original video image of the current time period to obtain at least one portrait detection information, each portrait detection information including a portrait ID, portrait frame coordinates and distance. In each time period, portrait detection, or face detection, is first performed based on the currently acquired original video image to determine the portrait in the original video image of the current time period. The front face or side face facing the camera or the display screen within a certain range can be determined as a portrait. For example Figure 5The original video screen shown in the figure can obtain the portrait detection information corresponding to each person through portrait detection, and one person corresponds to one portrait detection information, and the portrait detection information can be used in the form of an array. For example, the person in the middle is A, and the corresponding portrait detection information is A's portrait ID, the left frame coordinates of A's portrait frame, the left side of the right frame, the upper frame coordinates and the lower frame coordinates, and A's distance. Similarly, the person on the left is B, and the person on the right is C. ID is used as a portrait identifier. The portrait frame refers to a rectangular frame area that reflects the position and size of the portrait, and the portrait frame coordinates can include four coordinates corresponding to the four upper, lower, left and right sides respectively. The distance is the camera distance, which refers to the distance between the person and the camera in the actual physical space corresponding to the portrait. That is to say, in step 109, the distance between the user and the camera corresponding to each portrait in the original video screen of the current period collected by the camera is obtained. Figure 5 A in the middle is the closest, B on the right is the second closest, and C on the left is the farthest. It can be seen that the farther the distance, the smaller the size of the portrait frame.

[0117] like Figure 6 As shown, the above step 110 includes steps 200-304.

[0118] Step 200: remove the portrait detection information in the portrait frame coordinates in at least one portrait detection information that exceeds the boundary of the original video screen. For example, if the left frame coordinates of the portrait frame of a certain portrait detection information obtained in step 109 are less than 0, it means that the portrait frame exceeds the edge of the original video screen. Therefore, the portrait detection information is removed to avoid the problem of subsequent cropping of the screen exceeding the edge of the original screen.

[0119] Step 201: sort at least one portrait detection information in the original video screen of the current period in ascending order of distance to obtain sorted portrait detection information. The portrait detection information in the array may be sorted by a sorting algorithm such as a bubble algorithm, and the specific sorting algorithm will not be described in detail in this embodiment of the application. Figure 4 The obtained sorted portrait detection information is in the order of A, B, and C.

[0120] Step 202, in the sorted portrait detection information, remove the portrait detection information whose distance is 0.5m greater than the distance of the first portrait detection information, and obtain the portrait detection information after removal. The portrait corresponding to the user whose camera distance in the original video screen is closest is the first portrait, and the other portraits are the second portraits. The first portrait and all the second portraits with a preset distance whose difference in camera distance with the first portrait is ≤0.5m are portraits to be determined, and the portrait corresponding to the portrait detection information after removal in step 202 is the portrait to be confirmed. For example, the distance of A is closer to that of B, and the distance of C is 0.7m greater than that of A, so the portrait detection information corresponding to C is removed from the array, and the obtained portrait detection information after removal only includes the portrait detection information of A and B.

[0121] Step 203 , determining whether the number of people in the portrait detection information after removal is ≤ 2, if so, executing step 204 , if not, executing step 205 .

[0122] Step 204: Use the removed portrait detection information as valid portrait detection information. For example, in the above example, the removed portrait detection information only contains portrait detection information corresponding to two persons A and B. Then, the portrait detection information corresponding to two persons A and B is directly used as valid portrait detection information for subsequent use. In other words, if the number of portraits to be determined in the first area and the second area of ​​the original video screen of the current period is equal, all portraits to be determined in the original video screen of the current period are determined as target portraits.

[0123] Step 205, obtain the number of people whose portrait frame coordinate center points are located on the left side of the original video screen and the number of people whose portrait frame coordinate center points are located on the right side of the original video screen. The original video screen has 1920 pixels in the horizontal direction. If the portrait frame coordinate center point ≤ 960, it is located on the left side of the screen, and the number of people on the left side is +1. If the portrait frame coordinate center point > 960, it is located on the right side of the screen, and the number of people on the right side is +1. The area on the left side of the original video screen is the first area, and the area on the right side is the second area. Step 205 obtains the number of portraits to be confirmed in the first area and the number of portraits to be confirmed in the second area of ​​the original video screen of the current time period. In the subsequent process, the video screen will be displayed based on the distance between each user and the camera, the number of portraits to be confirmed in the first area, and the number of portraits to be confirmed in the second area.

[0124] Step 206 , determine whether the number of people on the left side is equal to the number of people on the right side, if so, execute step 204 , if not, execute step 207 .

[0125] Step 207: The portrait detection information corresponding to the larger one of the number of people on the left and the number of people on the right is used as the effective portrait detection information. Assuming that there are 3 people on the left side of the picture and 2 people on the right side of the picture, the portrait detection information corresponding to the 3 people on the left side of the picture is used as the effective portrait detection information for subsequent use. In other words, if the number of portraits to be determined in the first area and the second area of ​​the original video picture of the current period is not equal, all portraits to be determined in the first area and the second area of ​​the original video picture of the current period with the larger number of portraits to be determined are determined as target portraits.

[0126] The portrait in the valid portrait detection information determined in the above step 204 or step 207 is the target portrait. The process of steps 200 to 204 or steps 200 to 407 is to determine the target portrait in the portraits in the original video screen of the current time period. One valid portrait detection information corresponds to one portrait, so the number of valid portrait detection information is the number of target portraits. After the valid portrait detection information is determined in the above step 204 or step 207, step 301 is executed to obtain the first target area A of the current time period, step 302 is executed to obtain the second target area B of the current time period, and step 303 is executed to obtain the third target area C of the current time period. Figure 4 As shown, the first target area A refers to the smallest rectangular frame range of the portrait frame corresponding to all valid portrait detection information of the current time period, and the second target area B of the current time period is the frame after the first target area A of the current time period is expanded outward by a certain range, that is, the second target area B of the current time period is the enlarged and centered frame of the portrait frame corresponding to the valid portrait detection information of the current time period, and the second target area B refers to the final target area of ​​the current screen cropping. The third target area C is the actual cropping frame that gradually approaches the second target area B. The above-mentioned processes for obtaining the target areas are actually the processes for obtaining the coordinates of each target area, including the coordinates of the four left, right, upper and lower frames. In some cases, if the frame of the first target area A of the current time period after expanding outward by a certain range exceeds the edge of the original video screen, it is necessary to ensure that the second target area does not exceed the edge of the original video screen. At this time, the second target area B will cover the first target area A, but it does not necessarily have an obvious centering effect.

[0127] like Figure 7As shown, the above step 301 specifically includes steps 3011 to 3014. Step 3011, determine whether the number of valid portrait detection information is 0, if so, execute step 3012, if not, execute step 3013. Step 3012, use the boundary coordinate group of the original video screen of the current period as the coordinates of the first target area of ​​the current period. When it is determined that the number of valid portrait detection information is 0, it means that no portrait in the picture is detected in step 109, or after removing the portrait detection information beyond the boundary in step 200, there is no valid portrait detection information in the array. In this case, the boundary coordinates of the original video screen are directly used as the coordinates of the first target area. Step 3013, use the portrait frame coordinates of one of the valid portrait detection information as the coordinates of the first target area of ​​the current period. Step 3014, traverse other valid portrait detection information, if the coordinates of any edge of the portrait frame of other valid portrait detection information are outside the first target area of ​​the current period, assign the coordinates of the edge to the first target area of ​​the current period. For example, there are three portraits in the original video screen, and these three portraits are portraits corresponding to the portrait frames in the effective portrait detection information. The circular or elliptical shape is the portrait. In step 3013, first, the portrait frame coordinates corresponding to any portrait are used as the coordinates of the first target area A. Then, in step 3014, the effective portrait detection information corresponding to the second portrait is first traversed to determine that the left frame coordinates and the upper frame coordinates of the portrait frame corresponding to the effective portrait detection information are both outside the first target area of ​​the current period. Therefore, the coordinate frame coordinates and the upper frame coordinates of the upper portrait frame are assigned to the first target area of ​​the current period. Then, the effective portrait detection information corresponding to the third portrait is traversed to determine that the left frame coordinates of the portrait frame corresponding to the effective portrait detection information are outside the first target area of ​​the current period. Therefore, the left frame coordinates of the left portrait frame are assigned to the first target area of ​​the current period. In this way, the first target area covers the portrait frame corresponding to each effective portrait detection information by traversing, and finally the first target area of ​​the current period is the minimum rectangular frame covering the portrait frames of all effective portrait detection information, thereby achieving the acquisition of the first target area.

[0128] like Figure 8 As shown, the above step 302 includes 3021-3029.

[0129] Step 3021, among multiple preset resolution levels, determine the preset resolution level to which the first target area of ​​the current time period belongs. For example, multiple resolution levels are pre-set based on the resolution of the original video screen, the resolution of the original screen is 1920*1080, the first level is 1920*1080, the second level is 1760*990, the third level is 1600*900, the fourth level is 1440*810, and the fifth level is 1280*720. In order to make the final cropped screen have the same ratio as the original video screen to avoid the problem of screen stretching during cropping, the aspect ratio of each preset resolution level can be set to be equal to the aspect ratio of the original video screen, all of which are 16:9. If the resolution of the first target area in the current time period is 1880*1000, which is between 1920*1080 and 1760*990, it belongs to the first gear 1920*1080; if the resolution of the first target area in the current time period is 1880*880, which is between 1920*1080 and 1440*810, it also belongs to the first gear 1920*1080; if the resolution of the first target area in the current time period is 640*480, it is below 1280*720 and belongs to the fifth gear 1280*720.

[0130] Step 3022: determine whether the preset resolution level of the first target area in the previous time period is the same as the preset resolution level of the first target area in the current time period, if so, execute step 3023, if not, execute step 3024. If the current time period is the first cycle, the boundary coordinates of the original video screen can be used as the coordinates of the first target area in the previous time period.

[0131] Step 3023: determine whether the first target area of ​​the current period is covered by the second target area of ​​the previous period, if yes, execute step 3025, if no, execute step 3024. If the current period is the first cycle, the frame of the original video screen can be used as the second target area of ​​the previous period.

[0132] Step 3025: determine whether the amount of valid portrait detection information in the current time period is the same as the amount of valid portrait detection information in the previous time period. If so, execute step 3026; if not, execute step 3024.

[0133] Step 3026: The second target area of ​​the previous time period is used as the second target area of ​​the current time period. That is, if the number of people with valid portrait detection information in the picture has not changed compared with the last time, and the movement range of the people has not exceeded the previous gear and the range of the second target area, there is no need to calculate a new second target area, and the second target area of ​​the previous time period is still used to avoid frequent changes in the second target area due to small changes in the portrait frame, thereby improving the problem of picture jitter.

[0134] Step 3024: determine whether the coordinates of the first target area in the current period are the same as the boundary coordinates of the original video screen in the current period. If not, execute step 3027. If yes, execute step 3029. Here, whether the coordinates are the same refers to whether the coordinates of the four sides are the same. If they are the same, execute step 3028. If the coordinates of any one of the four sides are different, execute step 3027.

[0135] Step 3027, calculate the coordinates of the second target area of ​​the current period, that is, calculate the coordinates of the four sides of the second target area of ​​the current period, and the specific calculation process will be described in detail later. That is to say, if compared with the last time, there is a change in the amount of effective portrait detection information, a change in the preset resolution level, or the first target area exceeds the range of the second target area of ​​the previous period, and the first target area of ​​the current period is different from the boundary of the original picture, then it is necessary to calculate the coordinates of the second target area of ​​the current period.

[0136] Step 3028, determine whether the first target area of ​​the current time period is the same as the second target area of ​​the previous time period, if not, execute step 3027, if yes, execute step 3029.

[0137] Step 3029: The original video screen of the current period is used as the second target area of ​​the current period. That is, if the boundary of the first target area is the boundary of the original screen and is the same as the second target area of ​​the previous period, it is not necessary to calculate the coordinates of the second target area of ​​the current period, and it is only necessary to assign the boundary coordinates of the original screen to the second target area of ​​the current period.

[0138] like Fig. 9 As shown, the above step 3027 includes steps 401 to 414.

[0139] Step 401 , determine whether the left border coordinate of the first target area A in the current time period is ≤16, if so, execute step 402 , if not, execute step 403 .

[0140] Step 402, set the left frame coordinates of the second target area B in the current period to 0, and the right frame coordinates to W, where W is the width of the preset resolution level to which the first target area A in the current period belongs. Assume that the resolution of the first target area A in the current period is 640*480, and the preset resolution level to which it belongs is 1280*720, that is, W=1280. If the left frame coordinates of the first target area A are ≤16, it is considered that the first target area A is close to the left edge of the original picture or has exceeded the left edge, so the second target area B in the current period is directly set to the left side of the picture based on the width of the preset resolution level to which it belongs.

[0141] Step 403, determine whether the right frame coordinate of the first target area A in the current time period is ≥MW-16, if so, execute step 404, if not, execute step 405, MW is the width of the original video screen, that is, MW=1920.

[0142] Step 404: Set the left frame coordinates of the second target area B in the current period to MW-W=1920-1280=840, and the right frame coordinates to MW, i.e., 1920. If the right frame coordinates of the first target area A in the current period with a resolution of 640*480 are ≥1920-16, it is considered that the first target area A is close to the right edge of the original video screen or has exceeded the right edge, so the second target area B in the current period is directly set to the right side of the screen based on the width of the preset resolution level.

[0143] Step 405: Determine whether mleft-(W-mW) / 2≤0 is satisfied. If so, execute step 402; if not, execute step 406. Fig.10 As shown, mletf is the left frame coordinate of the first target area A in the current period, mright is the right frame coordinate of the first target area A in the current period, W=1280, mW is the width of the first target area A in the current period, that is, mW=mrght-mleft=640. (W-mW) / 2=(1280-640) / 2=320. If mleft-320≤0, it means that if the left frame coordinate of the first target area A directly shifted to the left based on the resolution level of 1280*720 is used as the left frame coordinate of the second target area, it will exceed the left edge of the original picture. Therefore, step 402 is executed to directly set the width of the second target area B in the current period to the left side of the picture based on the preset resolution level.

[0144] Step 406: Set the left frame coordinates of the second target area B in the current time period to mleft-(W-mW) / 2=mleft-320, and the right frame coordinates to mleft-(W-mW) / 2+W=mleft-320+1280=mleft+960. That is, if the first target area A in the current time period is far away from the left edge and the right edge of the original image, the left frame coordinates and the right frame coordinates of the second target area B in the current time period are set based on the position of the first target area A in the current time period and the width of the preset resolution level to which it belongs.

[0145] After step 406, step 407 is executed to determine whether the right frame coordinates of the second target area B in the current time period are ≥MW. If so, step 404 is executed, and if not, step 408 is executed. Since in step 406, only the left frame coordinates of the second target area B are ensured not to exceed the boundary of the original screen, the right frame coordinates of the second target area B are judged in step 407. If they are ≥1920, it may exceed the right edge of the original screen, so return to step 404 and directly set the second target area B in the current time period to the right side of the screen based on the width of the preset resolution gear. If the right frame coordinates of the second target area B are less than 1920, it means that it will not exceed the right edge of the original screen, so the subsequent process of determining the coordinates of the upper and lower frames can be continued.

[0146] After the above steps 402, 404 and 407 are determined to be negative, the setting of the left and right border coordinates of the second target area B in the current period is completed, and then step 408 is executed to continue setting the upper and lower border coordinates.

[0147] Step 408 , determine whether the upper border coordinate of the first target area A in the current time period is ≤9, if so, execute step 409 , if not, execute step 410 .

[0148] Step 409, set the upper frame coordinates of the second target area B in the current period to 0, and the lower frame coordinates to H, where H is the height of the preset resolution level to which the first target area A in the current period belongs, i.e., H = 720. If the upper frame coordinates of the first target area A in the current period are ≤ 9, it is considered that the first target area A is close to the upper edge of the original screen or has exceeded the upper edge, so the second target area B in the current period is directly set to the upper side of the screen based on the height of the preset resolution level to which it belongs.

[0149] Step 410, determine whether the lower frame coordinate of the first target area A in the current time period is ≥MH-9, if so, execute step 411, if not, execute step 412, MH is the height of the original video screen, that is, MH=1080.

[0150] Step 411, set the upper frame coordinates of the second target area B in the current period to MH-H=1080-720, and the lower frame coordinates to MH=1080. If the lower edge coordinates of the first target area A in the current period are greater than or equal to 1080-9, it is considered that the first target area A is close to the lower edge of the original picture or has exceeded the lower edge, so the second target area B in the current period is directly set at the lower side of the picture based on the height of the preset resolution level.

[0151] Step 412, determine whether mtop-(H-mH) / 2≤0 is satisfied. If so, execute step 409, if not, execute step 413. mtop is the upper frame coordinate of the first target area A in the current time period, mbottom is the lower frame coordinate of the first target area A in the current time period, H=720, mH is the height of the first target area A in the current time period, that is, mH=mbottom-mtop=480. (H-mH) / 2=(720-480) / 2=120. If mtop-120≤0, it means that if the upper frame coordinate of the first target area A is directly moved up based on the resolution level of 1280*720 as the upper frame coordinate of the second target area, it will exceed the upper edge of the original screen. Therefore, execute step 409 to directly set the second target area B in the current time period at the upper side of the screen based on the height of the preset resolution level.

[0152] Step 413, set the upper frame coordinates of the second target area B in the current time period to mtop-(H-mH) / 2=mtop-120, and the lower frame coordinates to mtop-(H-mH) / 2+H=mtop-120+720=mtop+600. That is, if the first target area A in the current time period is far away from the upper edge and the lower edge of the original picture, the upper frame coordinates and the lower frame coordinates of the second target area B in the current time period are set based on the position of the first target area A in the current time period and the height of the preset resolution level to which it belongs.

[0153] After step 413, step 414 is executed to determine whether the lower frame coordinates of the second target area B in the current time period are ≥MH. If so, step 409 is executed. If not, the subsequent step 303 is executed. Since in step 413, only the left frame coordinates of the second target area B are ensured not to exceed the edge of the original screen, the lower frame coordinates of the second target area B are judged in step 414. If they are ≥1080, it may exceed the lower edge of the original screen, so return to step 409 and directly set the second target area B in the current time period to the lower side of the screen based on the height of the preset resolution gear. If the lower frame coordinates of the second target area B are less than 1080, it means that it will not exceed the lower edge of the original screen, so the subsequent process of obtaining the third target area in the current time period can be continued, that is, step 303 is executed.

[0154] After the above steps 409, 411 and 414 are determined to be no, the setting of the left border coordinates, right border coordinates, upper border coordinates and lower border coordinates of the second target area B of the current time period is completed, that is, the process of obtaining the second target area B of the current time period is completed, and then step 303 is executed to continue to obtain the third target area of ​​the current time period.

[0155] like Fig.11 As shown, the above step 303 includes steps 501 to 508.

[0156] Step 501, obtain the left frame gap lg, the right frame gap rg, the upper frame gap tg and the lower frame gap bg. The gap here refers to the distance between the third target area C' of the previous period and the second target area B of the current period, the left frame gap lg refers to the distance between the left frames of the two, the right frame gap rg refers to the distance between the right frames of the two, the upper frame gap tg refers to the distance between the upper frames of the two, and the lower frame gap refers to the distance between the lower frames of the two. If the current period is the first cycle, the frame of the original video picture can be used as the third target area C' of the previous period.

[0157] Step 502, determine whether the sum of the left frame gap and the right frame gap lg+rg<width speed VW, and the sum of the upper frame gap and the lower frame gap rg+bg<height speed VH, if yes, execute step 503, if not, execute step 504. The width speed VW and the height speed VH can be preset values, for example, the width speed VW is 48 pixels and the height speed VH is 27 pixels. In order to make the ratio of the final cropped picture the same as the original video picture to avoid the problem of picture stretching during cropping, the ratio of the height speed VH to the height speed VH can be set to be equal to the width-to-length ratio of the original video picture, both of which are 16:9.

[0158] Step 503, the second target area B of the current period is used as the third target area of ​​the current period. That is to say, if the difference between the frame coordinates of the second target area B of the current period and the frame coordinates of the third target area C' of the previous period is within the preset range, it means that the two are close enough, so the coordinates of the second target area B of the current period can be directly used as the coordinates of the third target area C of the current period. If the difference between the coordinates of the second target area B of the current period and the coordinates of the third target area C' of the previous period is outside the preset range, it means that the two are not close enough and the buffer gradient process is still needed. In the subsequent processes 505 to 508, the coordinates of the third target area of ​​the current period are made closer to the coordinates of the second target area of ​​the current period relative to the coordinates of the third target area of ​​the previous period. In this way, the tracking of the target portrait can be achieved through the gradient animation effect.

[0159] Step 504, determine whether the second target area of ​​the current time period is enlarged or reduced relative to the third target area of ​​the previous time period. If it is reduced, execute step 505; if it is enlarged, execute step 506; if the sizes are the same, that is, neither enlarged nor reduced, execute step 506.

[0160] Step 505, reduce the third target area C' of the previous time period to obtain the third target area C of the current time period. The reduction here can be reduced by using the width speed VW and the height speed VH as the step size, that is, in multiple consecutive cycles, step 505 is executed, and the width of the third target area C' of the previous time period is reduced by 48 pixels and the height is reduced by 27 pixels in each cycle. It can be understood that the reduction step size may also be different from the above-mentioned VW and VH, but if the reduction ratio is maintained at 16:9, the cropping ratio can be guaranteed to be the same as the original picture. In addition, the reduction step size can be a fixed value or a dynamically changing value.

[0161] Step 506 , determine whether the position of the second target area B in the current time period is the same as the position of the third target area C′ in the previous time period, if not, execute step 507 , if yes, execute step 508 .

[0162] Step 507: Move the third target area C' of the previous period toward the direction close to the second target area B of the current period to obtain the third target area C of the current period. The step length of the movement may be a fixed value or a dynamically changing value.

[0163] Step 508: Enlarge the third target area C' of the previous time period to obtain the third target area C of the current time period. The logic of enlargement is similar to that of reduction. The width speed VW and the height speed VH can be used as step lengths for enlargement. The step length of enlargement can also be different from the above-mentioned VW and VH. It can be understood that the step length of enlargement can also be different from the above-mentioned VW and VH, but if the 16:9 ratio is maintained, the cropping ratio can be guaranteed to be the same as the original picture. In addition, the step length of enlargement can be a fixed value or a dynamically changing value.

[0164] After step 505, 507 or 508, the third target area C of the current period has been obtained, that is, the coordinates of the four sides of the third target area C have been obtained, so step 304 can be executed to crop the original video picture of the current period based on the third target area C of the current period. For example, if the size of the original video picture is 1080P, and the size of the third target area C of the current period is 1900*900, the cropping process is to adjust the third target area C to the size of 1080P. Finally, the picture of the third target area C will be displayed in the size of 1080P.

[0165] In the process of obtaining the third target area C, the aspect ratio is guaranteed to be the aspect ratio of the original video image, so as to avoid the problem of stretching and deformation of the final cropped image.

[0166] The processes of zooming in and zooming out are described below.

[0167] like Fig.12As shown, in the continuous multiple cycles t11 to t18, if the second target area B does not change, the size and position of the third target area C gradually approach the second target area B. The third target area is the actual cropping frame of the picture, and it is also a frame that may gradually change. The second target area B is the final cropping frame, and it is also the centered magnified frame of the portrait in the effective portrait detection information. The third target area C will gradually track the portrait in the effective portrait detection information in the continuous multiple cycles, that is, the third target area C will gradually approach the second target area B. If the coordinates of the third target area C' in the previous time period are close to a certain extent to the second target area B in the current time period, the coordinates of the second target area B are directly assigned to the third target area C in the current time period to achieve the acquisition of the third target area C. In the t11 period, in step 504, it is determined that the second target area B of the current period is smaller than the third target area C' of the previous period, so step 505 is executed, the center point of the third target area C' of the previous period remains unchanged, and the frame coordinates after the size is reduced are used as the frame coordinates of the third target area C of the current period, and then the screen is cropped and displayed with the third target area C; then entering the t12 period, the third target area C' of the previous period is still reduced as the third target area C of the current period; and so on, in the t11 to t15 periods, the third target area C' of the previous period is reduced as the third target area C of the current period; in the t16 period, in step 504, it is determined that the size of the second target area B of the current period is the same as that of the third target area C' of the previous period, that is, it has been reduced in place, so step 506 is executed to determine the size of the second target area B of the current period relative to the third target area C' of the previous period. The position of the third target area C' in the time period is different, that is, it has not been moved to the position. The position can be determined based on the center point coordinates of the two. Therefore, step 507 is executed to move the third target area C' of the previous time period to the direction close to the second target area B of the current time period to obtain the third target area C of the current time period; similarly, in the t17 period, the third target area C' of the previous time period is also moved to the direction close to the second target area B of the current time period to obtain the third target area C of the current time period; in the t18 period, in step 502, it is determined that the sum of the left frame gap and the right frame gap lg+rg<width speed VW, and the sum of the upper frame gap and the lower frame gap rg+bg<height speed VH, that is, it is determined that the size and position of the third target area C' of the previous time period and the second target area B of the current time period have been adjusted to the preset range, so the second target area B of the current time period is directly used as the third target area C of the current time period. In the process of t11 to t18, the process of the third target area C gradually changing to the second target area B in multiple cycles is realized, that is, the tracking of the portrait of the effective target is realized by animation.In addition, during the process of t11 to t18, the previous third target area needs to be reduced and moved. In the overall process, this is achieved by first reducing and then moving. This gradual change method can avoid the problem of the third target area C exceeding the display edge due to movement before reduction.

[0168] In addition, in multiple consecutive cycles t21 to t26, the third target area C presents a gradual manner of first moving and then enlarging. In the t21 cycle, it is determined in step 504 that the second target area B of the current time period is enlarged relative to the third target area C' of the previous time period, so step 506 is executed to determine that the position of the second target area B of the current time period is different from that of the third target area of ​​the previous time period, which can be specifically determined based on the coordinates of the center points of the two, so step 507 is executed to move the third target area C' of the previous time period to the right to approach the second target area B of the current time period, and the third target area C of the current time period is obtained after the movement; then enter the t22 cycle, similar logic is used to continue to move the third target area C' of the previous time period to the right to obtain the third target area C of the current time period; then enter the t23 cycle, similar logic is used to continue to move the third target area C' of the previous time period to the right to obtain the third target area C of the current time period; then enter the t24 cycle, in step 506, it is determined that the second target area B of the current time period is closer to the third target area B of the current time period. Since the position of the third target area of ​​the previous time period is the same, that is, it is moved to the right position, step 508 is executed to enlarge the third target area C' of the previous time period to obtain the third target area C of the current time period; then enter the t25 period, in step 506, it is determined that the position of the second target area B of the current time period is the same as that of the third target area of ​​the previous time period, and step 508 is executed to enlarge the third target area C' of the previous time period to obtain the third target area C of the current time period; in the t26 period, in step 502, it is determined that the sum of the left frame gap and the right frame gap lg+rg<width speed VW, and the sum of the upper frame gap and the lower frame gap rg+bg<height speed VH, that is, it is determined that the size and position of the third target area C' of the previous time period and the second target area B of the current time period have been adjusted to the preset range, so the second target area B of the current time period is directly used as the third target area C of the current time period. In the process of t21 to t26, it is realized by moving first and then enlarging. This gradual change method can avoid the problem that the third target area C exceeds the display edge due to enlarging before moving.

[0169] In a possible implementation, in the process of obtaining the third target area C of the current time period in the above steps 503, 505, 507 and 508, the relevant values ​​are multiplied by 10 and then calculated by integer int to obtain the third target area C of the current time period with the coordinate value magnified 10 times, where the relevant values ​​include the coordinates of the four edges and the step length of the movement. In the above step 304, the coordinate value of the third target area C of the current time period with the coordinate value magnified 10 times is divided by 10, and the original video screen of the current time period is cropped by floating point based on the restored third target area C of the current time period. For the calculation of the coordinate value of the third target area C, if the calculation is directly based on the floating point method, the error will be accumulated due to the gradual change of the third target area C, which may result in a large final cropping error. In the embodiment of the present application, an example is given for explanation. For example, the coordinates of the upper left corner of the third target area C in the previous period are (100, 100), the coordinates of the lower right corner are (420, 280), and the moving step length of the third target area C is (4.8, 2.7). These are all related values. First, these related values ​​are magnified by 10 times, that is, multiplied by 10, the coordinates of the upper left corner become (1000, 1000), the coordinates of the lower right corner become (4200, 2800), and the moving step length of the third target area C becomes ( 48,27), it can be seen that after zooming in 10 times, the calculation method of integer int can be used to calculate the third target area C after the move, that is, the third target area C of the current period is (1048,1027) and (4248,2827), and then the coordinates are divided by 10, which can be restored to the correct coordinate value to continue the subsequent floating point operation, that is, the coordinates of the third target area C of the current period are restored to (104.8,102.7) and (424.8,282.7). In this way, the error in the calculation process can be reduced and the accuracy can be improved.

[0170] It should be noted that the above description is only based on the application of the video processing method to a video call process, but the embodiment of the present application does not limit the application scenario of the video processing method. For example, in other possible implementations, the video processing method can also be applied to other scenarios that require video recording or video preview, such as video shooting scenarios of camera applications. In addition, the electronic device is a smart TV for example only. The electronic device can also be any electronic device such as a mobile phone, a tablet computer, a laptop computer or a car-mounted device. The camera can be the camera of the electronic device itself or an external camera.

[0171] The following describes the video processing method using some specific scenarios as examples.

[0172] In a possible implementation, Fig.13As shown, the above-mentioned display of the cropped picture includes: displaying the cropped picture in any one or any combination of the video call notification interface, the video phone dialing interface and the video call interface, and the cropped picture is that the picture of the third target area C is adjusted to a preset size.

[0173] In a possible implementation, Fig.13 As shown, when a video call is received, the above-mentioned video processing method is executed, including obtaining the original video screen captured by the camera, and displaying a video call notification interface, in which the cropped screen in the above step 304 is displayed, and the cropped screen is used as a preview screen for the video call; and / or, when making a video call, the original video screen captured by the camera is obtained, and a video call dialing interface is displayed, in which the cropped screen in the above step 304 is displayed, and the cropped screen is used as a preview screen for the video call; and / or, during a video call, the above-mentioned video processing method is executed, including periodically obtaining the original video screen captured by the camera, and displaying a video call interface, in which the cropped screen in the above step 304 is displayed, and the cropped screen is used as the screen for the video call.

[0174] It should be noted that the above description only takes the video passing scene as an example. The video processing method of the embodiment of the present application can be applied to other camera shooting scenes such as camera preview and camera recording.

[0175] In a possible implementation, Fig.14 As shown, in the first time period, the first video screen is displayed. In the first time period, the distance between the first user P1 and the camera is smaller than the distance between the second user P2 and the camera, and the distance between the first user P1 and the camera is smaller than the distance between the third user P3 and the camera. The first video screen includes the portrait of the first user P1. Steps 101 to 111 are performed in the first time period and each time period before the first time period. The process of performing the above steps in the first time period is described below. The original video screen is Fig.14 The larger rectangular frame in the image corresponds to the first video image. Fig.14 The picture corresponding to the smaller rectangular frame in the image is the picture corresponding to the second target area B. In fact, the first video screen picture should refer to the picture corresponding to the third target area. In this embodiment, the third target area is the same as the second target area B. Fig.14The original video screen shown in FIG. 1 is used to perform portrait detection based on the screen in step 108, and obtain three portrait detection information. The circle or ellipse in the figure represents a portrait, and the size of the portrait is positively correlated with the corresponding distance. In step 201, the portrait corresponding to the first user on the right is the portrait with the closest camera distance, which is the first portrait, and the corresponding portrait detection information is the first portrait detection information after sorting. Among them, the camera distance between the second user P2 and the third user P3 in the figure is relatively large, and the difference between the camera distance of the second user P2 and the camera distance of the first user P1 is greater than 0.5m, and the difference between the camera distance of the third user P3 and the camera distance of the first user P1 is greater than 0.5m. Therefore, in step 202, the portrait detection information corresponding to the second user P2 and the third user P3 is removed, and the portrait detection information after removal only corresponds to the first user P1, and the portrait corresponding to the first user P1 is the portrait to be determined. Then, in step 204, the portrait corresponding to the first user P1 is determined as the target portrait. Then, in steps 301 and 302, the first target area and the second target area B of the current time period are obtained based on the portrait corresponding to the first user P1. Fig.14 Only the second target area B is illustrated, and the second target area B covers the portrait corresponding to the first user P1. Then in step 303, the third target area of ​​the current time period is obtained based on the second target area B of the current time period and the third target area of ​​the previous time period. Assuming that the third target area of ​​the previous time period is the second target area B of the current time period, then in the current time period, the cropped picture includes the portrait corresponding to the first user P1, but does not include the portraits corresponding to the second user P2 and the third user P3; assuming that the third target area of ​​the previous time period is not the second target area B of the current time period, then the cropped picture will gradually become only including the portrait corresponding to the first user P1. The third area refers to the area in the actual physical space corresponding to the first area in the original video picture, and the fourth area refers to the area in the actual physical space corresponding to the second area in the original video picture. In the first time period, the first user P1 is located in the third area, and the second user P2 and the third user P3 are located in the fourth area. During the above steps 201 to 304, based on the fact that during the first time period, the distance between the first user P1 and the camera is less than the distance between the second user P2 and the camera, and the difference between the two is greater than 0.5m, and the distance between the first user P1 and the camera is less than the distance between the third user P3 and the camera, and the difference between the two is greater than 0.5m, the first video screen is displayed, that is, the screen with the first user P1 as the protagonist is displayed.

[0176] After the first user P1 moves away from the camera, the original video image changes from Fig.14 As shown Fig.15 As shown, in the second period, the second video screen is displayed, and the second video screen is Fig.15The second video screen includes the portraits of the second user P2 and the third user P3, as shown in the smaller rectangular box. After the first user P1 moves away from the camera, steps 101 to 111 are performed in the second time period. In step 201, the portrait corresponding to the first user P1 is still the portrait closest to the camera, which is the first portrait, and the corresponding portrait detection information is the first portrait detection information after sorting. After the first user P1 moves away from the camera, the difference between the camera distance of the second user P2 and the camera distance of the first user P1 is less than 0.5m, and the difference between the camera distance of the third user P3 and the camera distance of the first user P1 is less than 0.5m. Therefore, in step 202, the portrait detection information corresponding to the second user P2 and the third user P3 is not removed, and the portraits corresponding to the first user P1, the second user P2 and the third user P3 are portraits to be determined. Then, in step 207, the portraits corresponding to the second user P2 and the third user P3 in the first area on the left are determined as target portraits. Then, in steps 301 and 302, the first target area and the second target area B of the current time period are obtained based on the portraits corresponding to the second user P2 and the third user P3. Fig.15 Only the second target area B is illustrated, and the second target area B covers the portraits corresponding to the second user P2 and the third user P3. Then in step 303, the third target area of ​​the current time period is obtained based on the second target area B of the current time period and the third target area of ​​the previous time period, that is, in the second time period after the first user P1 moves away from the camera, the cropped picture includes the portraits corresponding to the second user P2 and the third user P3, but does not include the portrait corresponding to the first user P1. In the second time period, the positions of the portraits corresponding to the three users in the original video picture remain unchanged, but the distances between the three users and the camera are equal. Since the number of users in the fourth area is greater than the number of users in the third area, the picture with the second user P2 and the third user P3 in an area with a larger number of users as the protagonists.

[0177] In the process from the first time period to the second time period, the video display interface gradually changes from displaying the first video screen to displaying the second video screen. In each time period from the first time period to the second time period, steps 101 to 111 are performed, wherein it is assumed that the distance between the second user P2 and the camera and the distance between the third user P3 and the camera are equal and unchanged. In the process of the first user P1 moving away from the camera, the distance between the first user P1 and the camera gradually decreases. When it decreases to the difference between the distance between the second user P2 and the camera ≤ 0.5m, then in step 202, the portrait detection information corresponding to the second user P2 and the third user P3 is no longer removed, that is, the second target area B is composed of Fig.14 Became Fig.15 However, the sudden change of the second target area B does not cause a sudden change of the picture displayed on the video display interface. The picture displayed on the video display interface gradually changes from Fig.14 The second target area B in the Fig.15 of the second target area B.

[0178] like Fig.16 As shown, after the fourth user P4 moves into the camera capture range, the third video screen is displayed in the third time period. In the third time period, the distance between the fourth user P4 and the first user P1 is less than the distance between the second user P2 and the first user P1, that is, the distance between the fourth user P4 and the first user P1 is closer, while the distance between the second user P2 and the first user P1 is farther, and when the distance between the fourth user P4 and the second user P2 is greater than the distance between the second user P2 and the third user P3, that is, the distance between the fourth user P4 and the second user P2 is farther, while the distance between the second user P2 and the third user P3 is closer, the third video screen includes the portraits of the first user P1, the second user P2, the third user P3 and the fourth user P4. The distance in this paragraph does not refer to the camera distance, but to the distance between users. When the fourth user P4 is added to the camera capture range, steps 101 to 111 are executed in the third time period, and steps 101 to 111 are executed in each time period between the second time period and the third time period. Taking the third time period as an example, in step 201, the portrait corresponding to the first user P1 is still the portrait closest to the camera, which is the first portrait, and the corresponding portrait detection information is the first portrait detection information after sorting. When the fourth user P4 is added to the camera capture range, the difference in camera distance between other users and the first user P1 is less than 0.5m. Therefore, in step 202, the portrait detection information corresponding to the second user P2, the third user P3 and the fourth user P4 is not removed, and the portraits corresponding to the first user P1, the second user P2, the third user P3 and the fourth user P4 are portraits to be determined. Then, in step 204, the portraits corresponding to the first to fourth users are determined as target portraits, and then in steps 301 and 302, the first target area and the second target area B of the current time period are obtained based on the portraits corresponding to these four users. Fig.16 Only the second target area B is illustrated, and the second target area B covers the portraits corresponding to the four users. Then, in step 303, the third target area of ​​the current time period is obtained based on the second target area B of the current time period and the third target area of ​​the previous time period, that is, when the camera capture range is increased to include the fourth user P4 in multiple time periods, the cropped picture gradually changes from including the portraits corresponding to the second user P2 and the third user P3 to including the portraits corresponding to the four users.

[0179] When the second user P2, the third user P3 and the fourth user P4 leave the camera capture range and the first user P1 moves, the original video picture changes from Fig.16 As shown Fig.17As shown, in the fourth time period, the fourth video screen is displayed, and the fourth video screen includes the portrait of the first user P1. When the second user P2, the third user P3 and the fourth user P4 leave the camera capture range, steps 101 to 111 are performed in the fourth time period, and steps 101 to 111 are performed in each time period between the third time period and the fourth time period. Taking the fourth time period as an example, in step 201, the portrait of the first user P1 is determined as the target portrait in step 204, and then the first target area and the second target area B of the current time period are obtained based on the portrait of the first user P1 in steps 301 and 302. Fig.17 Only the second target area B is illustrated, and the second target area B covers the portrait of the first user P1. Then in step 303, the third target area of ​​the current time period is obtained based on the second target area B of the current time period and the third target area of ​​the previous time period, that is, in the fourth time period after the second user P2, the third user P3 and the fourth user P4 leave the camera capture range, the cropped picture becomes to include the portrait of the first user P1.

[0180] It can be seen that in multiple consecutive time periods, as the camera distance corresponding to the portraits in different areas of the original video screen changes, and as the target portrait changes, the size and position of the second target area B also change, and the third target area will continue to approach the second target area B. Therefore, the cropped image actually displayed will gradually track and transform to cover the target portrait. The protagonist is re-determined based on the number of people corresponding to the portraits in the video screen and the change in the camera distance, and the newly determined protagonist is tracked and enlarged through the cropped image, thereby highlighting the protagonist portrait in the video screen, achieving focus on the protagonist, and making it easier to see the protagonist portrait.

[0181] An embodiment of the present application also provides a video processing device, including: a first display module, used to display a first video screen in a first time period, in which the distance between the first user and the camera is smaller than the distance between the second user and the camera, and the distance between the first user and the camera is smaller than the distance between the third user and the camera, and the first video screen includes a portrait of the first user; a second display module, used to display a second video screen after the first user moves away from the camera, and the second video screen includes portraits of the second user and the third user; a third display module, used to display a third video screen after the fourth user moves into the capture range of the camera, the distance between the fourth user and the first user is smaller than the distance between the second user and the first user, and the distance between the fourth user and the second user is greater than the distance between the second user and the third user, and the third video screen includes portraits of the first user, the second user, the third user and the fourth user.

[0182] The video processing device can apply the video processing method in any of the above embodiments. The specific process and principle are the same as those in the above embodiments and will not be repeated here.

[0183] It should be understood that the division of the above video processing device is only a division of logical functions. In actual implementation, all or part of the modules can be integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, any one of the display modules can be a separately established processing element, or it can be integrated in the video processing device, such as being integrated in a certain chip of the video processing device. In addition, it can also be stored in the memory of the video processing device in the form of a program, and called and executed by a certain processing element of the video processing device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0184] For example, each display module may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a certain module above is implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0185] like Figure 1 As shown, an embodiment of the present application further provides an electronic device 100, wherein the internal memory 121 is used to store at least one instruction, and when the instruction is loaded and executed by the processor 110, the electronic device 100 executes the video processing method in any of the above embodiments.

[0186] The electronic devices involved in this application may be smart TVs, mobile phones, tablet computers, personal computers (PCs), personal digital assistants (PDAs), smart watches, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, vehicle-mounted devices, drone devices, smart cars, smart speakers, robots, smart glasses, and any other products.

[0187] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the video processing method in any of the above embodiments.

[0188] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0189] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or the second signal line can represent the cases of A existing alone, A and the second signal line existing simultaneously, and the second signal line existing alone. Wherein A and the second signal line can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, the second signal line, and the third signal line can represent: a, the second signal line, the third signal line, a - the second signal line, a - the third signal line, the second signal line - the third signal line, or a - the second signal line - the third signal line, where a, the second signal line, and the third signal line can be single or multiple.

[0190] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that: include: In a first time period, a first video screen is displayed. In the first time period, a distance between the first user and the camera is smaller than a distance between the second user and the camera, and a distance between the first user and the camera is smaller than a distance between the third user and the camera. The first video screen includes a portrait of the first user. After the first user moves away from the camera, displaying a second video screen, wherein the second video screen includes portraits of the second user and the third user; After the fourth user moves into the camera capture range, a third video screen is displayed, the distance between the fourth user and the first user is smaller than the distance between the second user and the first user, and the distance between the fourth user and the second user is larger than the distance between the second user and the third user, and the third video screen includes portraits of the first user, the second user, the third user and the fourth user.

2. The video processing method according to claim 1, characterized in that: In the first time period, the first user is located in a third area, and the second user and the third user are located in a fourth area; The displaying of the first video picture in the first time period includes: Based on the fact that the distance between the first user and the camera is smaller than the distance between the second user and the camera, and the distance between the first user and the camera is smaller than the distance between the third user and the camera within the first time period, the first video image is displayed.

3. The video processing method according to claim 2, characterized in that: After the first user moves away from the camera, displaying the second video picture includes: After the first user moves away from the camera, in the second time period, based on the fact that within the second time period, the distance between the first user and the camera is equal to the distance between the second user and the camera, the distance between the first user and the camera is equal to the distance between the third user and the camera, and the number of users in the fourth area is greater than the number of users in the third area, the second video screen is displayed.

4. The video processing method according to claim 3, characterized in that: During the period from the first time period to the second time period, the video display interface gradually changes from displaying the first video picture to displaying the second video picture.

5. The video processing method according to claim 1, characterized in that: The displaying of the first video picture in the first time period includes: Acquire a second target area including at least one human portrait in the original video picture collected by the camera during the first period of time; Based on the second target area of ​​the first time period and the third target area of ​​the last time period adjacent to the first time period, the third target area of ​​the first time period is acquired, the coordinates of the third target area of ​​the first time period are closer to the coordinates of the second target area of ​​the first time period than the coordinates of the third target area of ​​the last time period adjacent to the first time period, or the coordinates of the third target area of ​​the first time period are equal to the coordinates of the second target area of ​​the first time period; The original video picture captured by the camera in the first time period is cropped based on the third target area in the first time period to obtain a cropped first video picture, and the first video picture is displayed.

6. The video processing method according to claim 5, characterized in that: Also includes: Determine a target person portrait from the person portraits in the original video image captured by the camera during the first period of time, and obtain a first target area during the first period of time, wherein the first target area during the first period of time is a minimum rectangular frame including all the target person portraits; The step of acquiring a second target area including at least one human portrait from the original video image captured by the camera during the first time period includes: If the non-calculation condition is met, the second target area of ​​the previous time period adjacent to the first time period is used as the second target area of ​​the first time period; If the non-calculation condition is not met, obtaining a second target area including the first target area of ​​the first time period in the original video picture of the first time period, wherein a size of the second target area of ​​the first time period is larger than a size of the first target area of ​​the first time period; The non-computation condition includes at least one of the following or any combination thereof. When the non-computation condition includes multiple items, if any one of the multiple items is not satisfied, the non-computation condition is not satisfied, and if all of the multiple items are satisfied, the non-computation condition is satisfied: The preset resolution level to which the first target area in the previous time period adjacent to the first time period belongs is the same as the preset resolution level to which the first target area in the first time period belongs; The first target area of ​​the first time period is covered by the second target area of ​​the previous time period adjacent to the first time period; The number of the target human portraits in the first time period is equal to the number of the target human portraits in the previous time period adjacent to the first time period.

7. The video processing method according to claim 6, characterized in that: In the first time period, the first user is located in a third area, and the second user and the third user are located in a fourth area; The portrait corresponding to the user whose camera distance in the original video screen is closest is a first portrait, and the other portraits are second portraits, and all the second portraits whose difference in camera distance between the first portrait and the first portrait is less than or equal to a preset distance are portraits to be determined, and the original video screen includes a first area corresponding to the third area and a second area corresponding to the fourth area; Determining the target portrait from the portraits in the original video images captured by the camera during the first period of time includes: If the number of the to-be-determined portraits in the first area and the second area of ​​the original video image captured by the camera during the first period of time is not equal, all the to-be-determined portraits in the one with a larger number of the to-be-determined portraits in the first area and the second area of ​​the original video image captured by the camera during the first period of time are determined as target portraits; If the number of the to-be-determined portraits in the first area and the second area of ​​the original video image captured by the camera in the first period of time is equal, all the to-be-determined portraits in the original video image captured by the camera in the first period of time are determined as target portraits.

8. The video processing method according to claim 5, characterized in that: The process of acquiring the third target area of ​​the first time period based on the second target area of ​​the first time period and the third target area of ​​the previous time period adjacent to the first time period includes: If the second target area of ​​the first time period is smaller than the third target area of ​​the previous time period adjacent to the first time period, the third target area of ​​the previous time period adjacent to the first time period is reduced to obtain the third target area of ​​the first time period; If the second target area of ​​the first time period has the same size as the third target area of ​​the previous time period adjacent to the first time period, and the center coordinates of the second target area of ​​the first time period are different from the center coordinates of the third target area of ​​the previous time period adjacent to the first time period, the third target area of ​​the previous time period adjacent to the first time period is moved in a direction close to the second target area of ​​the first time period to obtain the third target area of ​​the first time period; If the size of the second target area of ​​the first time period is enlarged relative to the third target area of ​​the previous time period adjacent to the first time period, and the center coordinate position of the second target area of ​​the first time period is the same as the third target area of ​​the previous time period adjacent to the first time period, then the third target area of ​​the previous time period adjacent to the first time period is enlarged to obtain the third target area of ​​the first time period.

9. The video processing method according to claim 8, characterized in that: The process of reducing the third target area of ​​the previous time period adjacent to the first time period to obtain the third target area of ​​the first time period includes: reducing 10 times the coordinate value of the third target area of ​​the previous time period adjacent to the first time period by an integer int based on 10 times the step value to obtain 10 times the coordinate value of the third target area of ​​the first time period; The process of moving the third target area of ​​the previous time period adjacent to the first time period toward the direction close to the second target area of ​​the first time period to obtain the third target area of ​​the first time period comprises: moving 10 times the coordinate value of the third target area of ​​the previous time period adjacent to the first time period toward the direction close to 10 times the coordinate value of the second target area of ​​the first time period based on 10 times the step value by an integer int to obtain 10 times the coordinate value of the third target area of ​​the first time period; The process of enlarging the third target area of ​​the previous time period adjacent to the first time period to obtain the third target area of ​​the first time period includes: enlarging 10 times the coordinate value of the third target area of ​​the previous time period adjacent to the first time period based on 10 times the step value by an integer int to obtain 10 times the coordinate value of the third target area of ​​the first time period; The process of cropping the original video picture of the first time period based on the third target area of ​​the first time period to obtain the cropped picture includes: Divide 10 times the coordinate value of the third target area in the first time period by 10 to obtain the coordinate value of the third target area in the first time period, and crop the original video picture of the first time period based on the coordinate value of the third target area in the first time period in a floating point manner to obtain the cropped picture.

10. The video processing method according to claim 5, characterized in that: The width-to-length ratio of the third target area is the same as the width-to-length ratio of the original video picture.

11. The video processing method according to claim 1, characterized in that: Displaying the first video screen includes displaying the first video screen in at least one or any combination of a video incoming call notification interface, a video phone dialing interface, and a video call interface.

12. A video processing device, characterized in that: include: A first display module is used to display a first video picture in a first time period, during which a distance between a first user and the camera is smaller than a distance between a second user and the camera, and a distance between the first user and the camera is smaller than a distance between a third user and the camera, and the first video picture includes a portrait of the first user; A second display module, configured to display a second video image after the first user moves away from the camera, wherein the second video image includes portraits of the second user and the third user; The third display module is used to display a third video screen after a fourth user moves into the camera capture range, wherein the distance between the fourth user and the first user is smaller than the distance between the second user and the first user, and the distance between the fourth user and the second user is larger than the distance between the second user and the third user, and the third video screen includes portraits of the first user, the second user, the third user and the fourth user.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, the electronic device executes the video processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The method comprises a program or an instruction, and when the program or the instruction is run on a computer, the method according to any one of claims 1 to 11 is executed.

Images