An image processing method and apparatus
By calculating and adjusting the tilt angle of the target person in the image captured by the camera component, the problem of the overhead view in video conferencing was solved, enabling eye-level communication between the participants and improving the meeting experience.
Patent Information
- Application Number
- CN202510638849.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2026-11-17
AI Technical Summary
In video conferencing, the camera is often installed at a high height and tilted downwards, causing participants to appear to be looking down at the camera, which affects the face-to-face communication experience for both parties.
By acquiring images captured by the camera component, the tilt correction amount of the target person is calculated, and the viewing angle in the image is adjusted according to the correction amount to make it close to the eye level or frontal viewing angle. Image processing methods are used to adjust the viewing angle of the camera to correct the tilt angle of the target person.
It enabled participants to communicate at eye level or face to face, enhancing the video conferencing experience.
Smart Images

Figure CN123123733A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology
[0002] With increasing cross-regional collaboration, video conferencing has become the primary mode of meeting. Participants in their local meeting room can use conferencing terminals to communicate face-to-face with participants in other meeting rooms. For example, conferencing terminals can provide various views, including a director's close-up mode, which displays a close-up of the current speaker and outputs the speaker's audio data, thereby enabling face-to-face communication with participants in other meeting rooms.
[0003] Typically, cameras in meeting rooms (or conference rooms) are positioned in fixed locations and mounted at a high height to cover the entire room and ensure everyone is visible. Furthermore, cameras are usually tilted downwards to focus on specific areas, improving image quality. However, this high and downward tilt results in a significant angle in the captured image, causing participants to appear from a "top-down" perspective in the video. For example, speakers closer to the camera may be over-magnified, with their heads occupying a large portion of the frame, forcing other participants to view them from above. This prevents eye-to-eye communication and leads to a poor meeting experience. Summary of the Invention
[0004] This application provides an image processing method and apparatus for adjusting the perspective of objects in a video frame, so that the perspective of both parties in a video conference is close to the human eye's eye level, thereby improving the meeting experience.
[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:
[0006] Firstly, an image processing method is provided, which is applied to a conferencing terminal. For example, the method can be applied to a video conferencing terminal or a component of the video conferencing terminal (e.g., a camera); or to a circuit or chip responsible for computing / processing functions in the video conferencing (such as a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), neural processing unit (NPU), artificial intelligence (AI) processor, application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA)); or to a logic node, logic module, or software capable of implementing all or part of the functions of the video conferencing terminal. For ease of description, the following example illustrates a video conference between a user of a first conferencing terminal and a user of a second conferencing terminal, with the method provided in the first aspect applied to the first conferencing terminal, which includes a camera component.
[0007] The method includes: a first conference terminal acquiring a first image captured by a camera component; determining a first correction amount for the tilt angle of a first target person within the first image; adjusting the first included angle to a second included angle based on the first correction amount; obtaining and outputting a second image, wherein the second included angle is smaller than the first included angle, and the first included angle is greater than 0°. The first image includes at least one target person, which includes a first target person. The tilt angle of the first target person is the included angle between a first plane and a second plane where the first target person is located, and the second plane is a plane perpendicular to the viewing angle of the camera component.
[0008] In this method, when the target person (e.g., the first target person) in the first image captured by the camera component (e.g., a webcam) of the first conference terminal has a certain tilt angle, the user of the second conference terminal sees the first target person's viewpoint as not at eye level or facing forward, resulting in a poor video conferencing experience. This method can correct the tilt angle of the first target person to make their viewpoint closer to eye level or facing forward. For example, a correction amount (e.g., a first correction amount) for the tilt angle of the first target person can be calculated, and the tilt angle of the first target person can be adjusted from a first angle to a second angle, where the second angle is smaller than the first angle. Thus, the tilt angle of the first target person in the corrected first image (i.e., the second image) becomes smaller, or even zero, making the first target person's viewpoint closer to eye level or facing forward. In this way, the user of the second conference terminal can see the image of the first target person looking directly at the camera on the first conference terminal side, thereby enabling both parties to communicate at eye level and improving the video conferencing experience.
[0009] In one design, determining a first correction amount for the tilt angle of a first target person includes: acquiring at least one parameter, and determining the first correction amount based on the at least one parameter and the correspondence between the at least one parameter and the correction angle. The at least one parameter may include one or more of the following: the horizontal distance between the mapped position of a first reference point in the first image in the first space and the camera component; the vertical distance between the mapped position of a second reference point in the first image in the first space and the camera component; the height of the camera component above the ground; or the angle between the viewing direction of the camera component and the horizontal plane.
[0010] For example, the above list of parameters that affect the tilt angle of the first target person can be used to determine the first correction amount based on one or more of the above parameters in order to obtain a more suitable correction amount so that after the tilt angle of the first target person is adjusted, the first target person's perspective is either facing forward or looking straight ahead, thus improving the meeting experience.
[0011] In one design, determining a first correction amount for the tilt angle of a first target person includes: determining a first correction amount when the horizontal distance between a first reference point and a camera component is less than or equal to a preset threshold; or, at least one target person includes a second target person, and the first correction amount is determined when the second target person is detected to have moved out of the shooting range of the camera component and the horizontal distance from the second target person to the camera component is less than or equal to a preset threshold; or, the first correction amount is determined when a third target person is detected to have entered the shooting range of the camera component and the horizontal distance from the third target person to the camera component is less than or equal to a preset threshold.
[0012] In some scenarios, the tilt angle of the primary target person is small, having little impact on the video conferencing experience. For example, when the horizontal distance between the primary reference point and the camera module is large, the tilt angle of the primary target person is small, possibly even zero, and their viewpoint is close to eye level or facing directly at them. Conversely, when the horizontal distance between the primary reference point and the camera module is less than or equal to a preset threshold, the tilt angle of the primary target person is large, affecting the meeting experience. In this case, determining the first correction amount can reduce unnecessary correction calculations, thereby reducing the power consumption of the conferencing terminal. For another example, the entry of a new target person into the camera module's field of view or the departure of an existing target person may affect the tilt angle of the primary target person. In this case, the first correction amount is determined to reduce unnecessary correction calculations, thereby reducing the power consumption of the conferencing terminal.
[0013] In one design, it is assumed that at least one target person is the first target person, the first reference point is the center position of the first target person, and the second reference point is the center position of the first target person; or, at least one target person is multiple target persons, the first reference point is determined based on the horizontal distance between the multiple target persons and the camera component, and the second reference point is determined based on the vertical distance between the multiple target persons and the camera component.
[0014] In some scenarios, there may be one or more target individuals within the camera's field of view. When multiple target individuals appear in the first image captured by the camera, it is desirable to minimize the tilt angle of each target individual or a few key target individuals. In this case, the first and second reference points used to calculate the tilt angle correction for each target individual also take into account the positions of multiple target individuals, in order to minimize the tilt angle of each target individual or a few key target individuals and improve the video conferencing experience.
[0015] In one design, adjusting a first included angle to a second included angle based on a first correction amount includes: adjusting the first included angle to the second included angle through at least two adjustments based on at least one step size and the first correction amount. The angle after the at least two adjustments equals the first correction amount.
[0016] For example, the first angle can be adjusted to the second angle in one adjustment, or it can be adjusted to the second angle in multiple adjustments. By adjusting the tilt angle of the first target person multiple times, with each adjustment being a small increment, a smooth transition of the first target person in the image can be achieved, improving the visual effect.
[0017] In one design, the first image is an image frame in either a director's close-up mode or a director's even-sharing mode; or, the first image is an image frame during the switching of multiple view modes, including a director's close-up mode and a director's even-sharing mode. The image processing method provided in this application is applicable to director scenarios.
[0018] Secondly, an image processing apparatus is provided, which is applied to a conference terminal and can be used to perform the method described in the first aspect and any of its designs. The beneficial effects can be found in the relevant description of the first aspect and will not be repeated here.
[0019] In one possible design, the device includes corresponding means, modules, or units for performing the method described in the first aspect and any of its designs. These means or units can be implemented in software, hardware, or a combination of both. For example, the device includes: an acquisition unit (sometimes also called an acquisition module), a determination unit (sometimes also called a determination module), and an adjustment unit (sometimes also called an adjustment module). The acquisition unit acquires a first image captured by the camera assembly, the first image including at least one target person, the at least one target person including a first target person; the determination unit determines a first correction amount for the tilt angle of the first target person, the tilt angle being a first angle between a first plane and a second plane where the first target person is located, the second plane being a plane perpendicular to the viewing angle of the camera assembly; the adjustment unit adjusts the first angle to a second angle according to the first correction amount, obtains and outputs a second image, the second angle being smaller than the first angle.
[0020] Optionally, the acquisition unit, determination unit, and adjustment unit can be three functional units divided within the processor. This application does not limit the specific type of processor; the processor can be a CPU, or other specific integrated circuits, such as a system-on-chip (SoC) chip, a system-in-package (SIP) chip, a DSP, GPU, NPU, AI processor, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0021] Thirdly, this application provides an image processing apparatus, including one or more processors and a memory coupled to each other. The memory stores part or all of a computer program (also referred to as code or instructions) necessary for implementing the functions involved in the first aspect and any of its designs. One or more processors can execute the computer program (or code or instructions), and when the computer program (or code or instructions) is executed, the apparatus implements the methods in the first aspect and any of its designs. This application does not limit the specific type of processor. For example, the processor can be a CPU, or other specific integrated circuits. As another example, the processor can be other general-purpose processors, GPUs, NPUs, AI processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0022] In one possible design, the memory is located outside the image processing device, or inside the communication processing device, or the processor and memory are integrated together.
[0023] Fourthly, a chip system is provided, including a processor and potentially a communication interface for implementing the methods of the first aspect and its various designs. Optionally, the chip system also includes a memory. The memory is used to store computer programs (also referred to as code or instructions). The processor is used to retrieve and execute the computer programs from the memory, causing a device equipped with the chip system to perform the methods of the first aspect and its various designs.
[0024] Fifthly, a computer-readable storage medium is provided, which stores a computer program or readable instructions, such that when a computer reads and executes the computer program or readable instructions, the computer performs the method described in the first aspect and any of its designs. Optionally, the storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).
[0025] Sixthly, a computer program product is provided, which, when read and executed, causes a computer to perform the method described in the first aspect and any of its embodiments. The computer program product may be a software installation package, which can be downloaded and executed on a computing device when the method provided in the first aspect or any possible implementation thereof is required.
[0026] The beneficial effects of the second to sixth aspects mentioned above can be referenced to the beneficial effects of the first aspect and any of its designs, and will not be repeated here. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of a video conferencing scenario;
[0028] Figures 2A-2B This is a schematic diagram of several broadcast view modes;
[0029] Figures 3A-3B Here are some schematic diagrams illustrating image distortion;
[0030] Figure 4 This is a diagram showing the target figure in the image at an angle.
[0031] Figure 5 This is a schematic diagram illustrating how a new perspective can be altered based on a novel perspective synthesis technique.
[0032] Figure 6 This is a schematic diagram of the architecture of a system applicable to the embodiments of this application;
[0033] Figure 7 This is a schematic diagram of a conference terminal.
[0034] Figure 8 This is a schematic diagram of another structure for a conference terminal;
[0035] Figure 9 This is a schematic flowchart of an image processing method provided in an embodiment of this application;
[0036] Figure 10 A diagram showing the angle of inclination for the primary target.
[0037] Figure 11 A schematic diagram for adjusting the tilt angle of the primary target;
[0038] Figure 12 Another diagram illustrating the adjustment of the tilt angle of the primary target figure;
[0039] Figure 13 Another schematic flowchart of the image processing method provided in the embodiments of this application;
[0040] Figure 14 A diagram illustrating the adjustment of the tilt angle of a target character in a scene with multiple target characters;
[0041] Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Detailed Implementation
[0042] In the embodiments of this application, the number of nouns, unless otherwise specified, refers to "singular nouns or plural nouns," that is, "one or more." "At least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A / B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. For example, A / B means: A or B. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and / or c means the following combination: a exists alone, b exists alone, c exists alone, a and b exist simultaneously, a and c exist simultaneously, b and c exist simultaneously, or a, b, and c exist simultaneously, where a, b, and c can be single or multiple.
[0043] In the embodiments of this application, "when," "if," and "if" all refer to the device taking corresponding actions under certain objective circumstances, not to a time limit, nor to requiring the device to perform a judgment action, nor implying any other limitations. Unless otherwise specified, "if" and "if" are interchangeable, and "when" is interchangeable with "in the case of," "if" / "if." Words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, "of," "corresponding (relevant)," and "corresponding" can sometimes be used interchangeably; it should be noted that their intended meanings are consistent unless their distinction is emphasized.
[0044] To facilitate understanding of the solutions provided in the embodiments of this application, the relevant terms and concepts involved in the embodiments of this application will be introduced first.
[0045] (1) Video conferencing
[0046] Video conferencing refers to a meeting that allows participants from different geographical locations to communicate face-to-face through a conferencing terminal. Through video conferencing, each participant can hear the audio from other meeting rooms and see video feeds of other participants, such as their appearance, actions, and facial expressions. They can also send electronic presentations.
[0047] See Figure 1This illustrates a video conferencing scenario where participants in meeting room #1 and meeting room #2 can conduct a remote video conference via conference terminals #1 and #2. During the video conference, conference terminal #1 collects audio and video data #1 from participants in meeting room #1 (e.g., participant #A) and sends it to conference terminal #2. Conference terminal #2 then outputs the audio and video data #1 (e.g., displays a video image), enabling communication between participants in meeting room #2 and those in meeting room #1 via conference terminal #2. Similarly, conference terminal #2 can collect audio and video data #2 from participants in meeting room #2 (e.g., participant #B) and send it to conference terminal #1. Conference terminal #1 then outputs the audio and video data #2, enabling communication between participants in meeting room #1 and those in meeting room #2 via conference terminal #1. It should be understood that the audio and video data includes audio data and / or video data.
[0048] (2) View modes of video conferencing
[0049] The conferencing terminal can display the video feed from participating video conferences. It offers various video view modes; some common view modes are illustrated below.
[0050] (2-1) Director's Close-up Mode: This mode focuses the camera on a specific participant (e.g., the current speaker), displaying a close-up of them to maximize their screen size and highlight key information or details such as facial expressions. When a participant in the panoramic view begins to speak, the display adjusts to a close-up of that speaker. If the speaker changes, the display adjusts to a close-up of the new speaker. Optionally, when no one is speaking, the display can be switched to a panoramic view.
[0051] See Figure 2A This shows one view of the director's close-up mode. (Continued...) Figure 1 For example, Figure 2A Take the display screen of conference terminal #1 as an example. Figure 2A In the interface shown in conference terminal #1, the main interface in the middle displays the video feed of one of the meeting rooms (e.g., meeting room #2) (e.g., the video feed of participant #B).
[0052] There are no restrictions on the specific name of the director's close-up mode; for example, it can also be called speaker positioning. Speaker positioning can be achieved through sound source localization (SSL). SSL refers to using multiple microphones at different locations in the environment to measure sound signals. Since the sound signals arrive at each microphone with varying degrees of delay, algorithms are used to process the measured sound signals to obtain the direction of arrival (including azimuth and pitch angles) and distance of the sound source relative to the microphones, thereby determining the location of the sound source in space.
[0053] Based on the director's close-up, the close-up shot will track and move in real time as the current speaker moves.
[0054] (2-2) Director's Even Distribution Mode: This mode can stitch together close-up shots of multiple participants in the conference room into a single image and present it.
[0055] Compared to the directing close-up mode, the directing equal-screen mode can also be considered as a stitching together of close-ups of multiple speakers, suitable for free discussion scenarios. The specific name for the directing equal-screen mode is not limited; for example, it can also be called the directing dialogue mode. When multiple figures are detected in the panoramic view, the speaker and active participants can be identified based on localization technology. An adaptive split-screen algorithm is then used to automatically generate close-ups of multiple figures according to their spatial distribution, ensuring that multiple figures are displayed on the screen in a more reasonable proportion.
[0056] See Figure 2B This shows one view of the director's close-up mode. (Continued...) Figure 1 For example, Figure 2B Take the display screen of conference terminal #2 as an example. Figure 2B In the interface shown in conference terminal #2, the main interface in the middle displays the video feed of one of the meeting rooms (e.g., meeting room #1).
[0057] In some scenarios, conference terminals may switch between multiple view modes, requiring a smooth transition from one view mode to another, also known as broadcast smoothing. In broadcast smoothing, the transition from one view mode's screen (e.g., the first screen) to another view mode's screen (e.g., the second screen) is not a direct switch between two frames, but a smooth transition process. In other words, in broadcast smoothing, there is a transition frame between the two frames before and after the switch; the transition frame outputs the intermediate screen between the before and after the switch.
[0058] (3) Intelligent Framing / Auto Framing: Utilizing image recognition technology and intelligent algorithms, the framing range and composition of the image acquisition device are automatically adjusted based on information such as the meeting scene, number of participants, and their locations to present the best meeting perspective. For example, the framing range may include the speaker's surrounding environment to achieve a close-up mode for broadcasting. In addition, intelligent framing may also have features such as evenly distributing the image, enabling an evenly distributed mode for broadcasting.
[0059] (4) Depth estimation
[0060] Depth estimation is a concept in computer vision that aims to infer the distance or depth information between objects in a scene and the observer using visual data (such as images or videos). For example, depth estimation can estimate the distance of each pixel in a two-dimensional image relative to the shooting source. There are no restrictions on the specific implementation of depth estimation; for example, it can employ methods based on stereo vision, monocular vision, structured light, or time-of-flight (ToF).
[0061] (5) Image distortion
[0062] Image distortion refers to the phenomenon where the captured image deviates from the geometry of the actual scene due to the optical characteristics of the camera lens or assembly errors. Image distortion is generally divided into radial distortion and tangential distortion.
[0063] Radial distortion is primarily caused by the optical characteristics of a lens. As light passes through the lens, the farther away from the lens center, the greater the degree of refraction and bending, resulting in radial distortion of objects in the image. Radial distortion specifically manifests as barrel distortion and pincushion distortion. For example... Figure 3A As shown, barrel distortion causes straight lines in an image to bend outwards, resembling the shape of a barrel; pincushion distortion causes straight lines to bend inwards, resembling a pincushion.
[0064] Tangential distortion is typically caused by factors such as the lens not being parallel to the image sensor plane or manufacturing defects in the lens itself. Tangential distortion causes objects in an image to shift and deform along the tangential direction, resulting in changes to the image's shape and proportions. For example... Figure 3B As shown, tangential distortion makes an originally square object appear as a trapezoid.
[0065] (6) Distortion correction
[0066] Distortion correction is a technique in image processing used to correct image distortions caused by various factors, aiming to restore the true form of a scene. There are various methods of distortion correction, including model-based methods such as traditional camera models and polynomial fitting, as well as machine / deep learning-based distortion correction methods. Distortion correction essentially changes the way the image is projected.
[0067] (7) AI models and AI functions
[0068] In this embodiment, the AI model can be deployed on an electronic device with specific image processing capabilities, such as a conference terminal or a camera. This electronic device can deploy one or more AI models to implement corresponding AI functions. The AI model can achieve different functions depending on its parameter configuration. The AI model can be configured based on one or more of the following parameters: structural parameters (e.g., at least one of the following: number of neural network layers, neural network width, inter-layer connections, neuron weights, neuron activation function, or bias in the activation function), input parameters (e.g., type and / or dimension of input parameters), or output parameters (e.g., type and / or dimension of output parameters). The bias in the activation function can also be referred to as the bias of the neural network.
[0069] In one example, the neural network mentioned above could be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), or a generative adversarial network (GAN).
[0070] The following description, in conjunction with the accompanying drawings, describes some aspects related to the embodiments of this application.
[0071] Video conferencing allows participants located in different geographical areas to communicate face-to-face. However, cameras in a typical venue (or conference room) are usually positioned in a fixed location and mounted at a high height to cover the entire room and ensure everyone is in the frame. Furthermore, cameras are generally tilted downwards to focus on a specific area and improve image quality. However, this fixed camera position results in different viewing angles depending on the distance between the camera and the person. The closer the person is to the camera, the higher the camera is, and the greater the camera's tilt angle, the more likely it is to capture the top of the person's head. This leads to a large tilt angle in the captured image, causing participants to appear from a "top-down" perspective in the video. For example, a speaker closer to the camera may be over-magnified, with their top of their head occupying a large proportion of the frame, causing other participants to face the speaker from a "top-down" perspective. This prevents participants from communicating at eye level, resulting in a poor video conferencing experience.
[0072] like Figure 4 As shown, from Figure 4 As can be seen in (a) above, the perspective of the person in the image is not at eye level. To improve the video conferencing experience, the image of the person in the image can be corrected. As mentioned earlier, image distortion correction actually changes the way the image is projected, but it does not change the viewing angle of the image. For example, using image distortion correction... Figure 4 The image shown in (a) can be obtained as follows: Figure 4 The image shown in (b) shows that, compared to... Figure 4 In (a) of the above, Figure 4 In (b), the distortion of the figures is reduced, resulting in a better overall visual experience. Figure 4 In the image shown in (b), the perspective of the character does not change.
[0073] New perspective synthesis technology, as one method to address image perspective issues, can correct the image's viewpoint. However, this technology achieves its purpose by adjusting the camera's position and pose, aiming to unify the entire image from a single perspective. For example... Figure 5 As shown, Figure 5 Images (a), (b), and (c) show the images before and after each adjustment of the camera's position and pose, respectively. It can be seen that the viewing angle can be adjusted by changing the camera's position and pose.
[0074] However, in meeting scenarios, the position and pose of the camera are usually fixed, making it impossible to change the viewing angle of objects in the image based on new perspective synthesis technology.
[0075] Therefore, the solution provided in this application embodiment is as follows. In this application embodiment, if the perspective of the target person in the image obtained by the first camera is not at eye level or frontal angle (e.g., looking up or down), then a correction amount for the target person's perspective is calculated. Based on this correction amount, the perspective of the target person is corrected so that the perspective of the target person in the corrected image is at eye level or frontal angle. The corrected image is sent to the second conference terminal, which displays the corrected image. The user of the second conference terminal can see the image of the user on the first conference terminal looking directly at the camera, thereby achieving eye-level communication and improving the video conferencing experience.
[0076] The image processing method provided in this application can be applied to a variety of application scenarios, as illustrated below.
[0077] Application Scenario 1: Video conferencing scenario.
[0078] Both the first conference terminal located in meeting room #1 and the second conference terminal located in meeting room #2 are equipped with video conferencing applications. User A in meeting room #1 uses the first conference terminal to conduct a video conference with User B in meeting room #2 using the second conference terminal. For example, the camera on the first conference terminal captures a first image of User A, which includes a facial image of User A. The facial image in the first image may be viewed from above. The first conference terminal can use the image processing method provided in this application embodiment to correct the viewing angle of the facial image in the first image. After the viewing angle is corrected, the eyes in the facial image are looking directly at the camera of the first conference terminal. The corrected image is then sent to the second conference terminal. In this way, the image received by User B's second conference terminal is an image of User A looking directly at the camera. Similarly, User B's second conference terminal can also use the same method to process the image. In this way, the image received by User A's first conference terminal is an image of User B looking directly at the camera, enabling User A and User B to communicate at eye level, thus improving the user experience.
[0079] Optionally, after the first conference terminal runs the conference app and enters shooting or recording mode, it can automatically enter the image tilt correction mode. Alternatively, the user can specify to enter the image tilt correction mode. In the image tilt correction mode, the first conference terminal can activate the image viewing angle correction method provided in this application embodiment. There are several ways for the user to specify to enter the image tilt correction mode. For example, the first conference terminal enters the image tilt correction mode when it detects a specific operation (e.g., a double-click operation within the viewfinder); or, the first conference terminal displays an icon for the image tilt correction mode when it detects a "More" button in a specific area, and enters the image tilt correction mode when an operation targeting that icon is detected.
[0080] Application Scenario 2: Live Streaming Scenario.
[0081] In application scenario 2, the electronic device can be a live streaming terminal, such as a mobile phone, tablet computer, or laptop computer. In the live streaming scenario, the live streaming environment includes the live streaming terminal and at least one live streaming object. During the live streaming process, the live streaming terminal acquires the target image captured by the camera component and collects the sound signal emitted by the sound source object in the live streaming environment through the microphone. The video image can be presented in either a director's close-up mode or a director's even-sharing mode, and this video image includes the target object. Optionally, before outputting the video image, if it is found that the target object's viewing angle in the image is not at eye level or facing forward, the image viewing angle correction method provided in this application embodiment can be used to correct the viewing angle of the target object in the image. After the viewing angle correction, the target object is facing the camera of the live streaming terminal.
[0082] Optionally, the electronic device is, for example, a mobile phone. When the mobile phone runs a live streaming app and enters shooting or recording mode, it can automatically enter image tilt correction mode. Alternatively, the user can specify to enter image tilt correction mode. In image tilt correction mode, the electronic device can activate the image viewing angle correction method provided in this application embodiment. There are several ways for the user to specify to enter image tilt correction mode. For example, the electronic device enters image tilt correction mode when it detects a specific operation (e.g., a double-click operation within the viewfinder); or, the electronic device displays an icon for image tilt correction mode when it detects a "More" button in a specific area, and enters image tilt correction mode when an operation targeting that icon is detected.
[0083] Application Scenario 3: Program Recording Scenario.
[0084] The program recording scenario may include multiple subjects being filmed. Electronic devices acquire camera components to capture images of these subjects, obtaining program video. During recording, the electronic devices can acquire target images of the recording environment and collect sound signals emitted by sound source objects in the recording environment via microphones. Based on the sound signals, the sound source object can be located from among the multiple subjects included in the target image, thereby performing directing operations on the sound source object. Optionally, before outputting the video image, if it is found that the target object's viewing angle within the image is not at eye level or facing forward, the image viewing angle correction method provided in this application embodiment can be used to correct the viewing angle of the target object in the image, so that the corrected target object is facing the camera of the live broadcast terminal.
[0085] Optionally, the electronic device, such as a camera, can automatically enter image tilt correction mode after entering shooting mode or recording mode, or the user can specify to enter image tilt correction mode. In image correction mode, the camera can activate the image viewing angle correction method provided in the embodiments of this application. There are several ways for the user to specify to enter image tilt correction mode. For example, the camera enters image tilt correction mode when it detects a specific operation (e.g., a double-click operation within the viewfinder); or, the camera displays an icon for image correction mode when it detects a "More" button in a specific area, and enters image tilt correction mode when an operation targeting that icon is detected.
[0086] The above application scenarios are merely illustrative examples, and the specific application scenarios of the solutions provided in the embodiments of this application are not limited. For example, the solutions in the embodiments of this application can also be applied to any scenario that requires image perspective correction, such as the creation of emoticons (e.g., the creation of emoticons in WeChat), the shooting of short videos on Douyin, Weibo, Xiaohongshu, etc., etc., which will not be listed one by one here.
[0087] For ease of description, the image processing method provided in the embodiments of this application is applied to video conferencing as an example, such as the directing mode of video conferencing.
[0088] As an example, Figure 6 The following diagram illustrates the architecture of a system to which embodiments of this application apply. Figure 6 The architecture shown includes conference terminals 10 (such as the first conference terminal to the third conference terminal) and servers (such as AI server 20 and media server 30).
[0089] Conference terminal 10 is deployed in a meeting room, allowing participants in different meeting rooms to communicate face-to-face based on the deployed conference terminal 10. The meeting room includes the capture range of the video capture device (or image capture device) associated with the conference terminal 10. The capture range of the video capture device refers to the set of spatial areas that the video capture device can capture. The video capture device associated with the conference terminal 10 refers to a video capture device capable of data interaction with the conference terminal 10, such as a video capture device installed on the conference terminal 10, and / or a video capture device that is relatively independently set up but capable of communicating with the conference terminal 10. The video capture device is, for example, a camera, camcorder, or video camera. Optionally, the video capture device associated with the conference terminal 10 includes a video capture device installed on the conference terminal 10, and includes a second video capture device that is independently set up relative to the conference terminal 10. In this case, the meeting room where the conference terminal 10 is located may include the capture range of the first video capture device and the capture range of the second video capture device. It should be noted that there can be multiple conference terminals 10 in a meeting room; this application uses one conference terminal 10 per meeting room as an example. The first meeting room may include one or more objects, such as participants, a blackboard, a conference table, etc.
[0090] A server can be a physical server or a virtual server running on a computer device. There can be one or more servers. For example, a server can be a single server or a server node in a server cluster. Servers can be deployed in a distributed or centralized manner. For example, a server can be centrally deployed as a single unit, or multiple servers can be distributed according to different functions. Figure 6 The architecture shown is illustrated using a distributed deployment of servers with different functions as an example. Figure 6 The servers shown include an AI server 20 and a media server 30. The AI server 20 can be used to implement AI functions, and the media server 30 is used to transmit media streams. Optionally, the system to which this application embodiment applies may not include the AI server 20 or the media server 30. For example, if the conference terminal 10 has AI functions, the system may not include the AI server 20; if the conference terminals 10 can transmit media streams to each other without going through the media server 30, the system may not include the media server 30.
[0091] The image processing method provided in this application can be applied to electronic devices with image processing capabilities, such as cameras or the aforementioned conference terminal 10. Optionally, the image processing method provided in this application can also be applied to media servers. For ease of description, the following example uses the image processing method provided in this application applied to a conference terminal.
[0092] Conference terminal 10 can acquire media streams, which may include one or more of the following: video stream, audio stream, image information, text information, bullet screen information, information from the shared interface, or screen projection information. Conference terminal 10 can capture local media streams (e.g., those from the conference room where conference terminal 10 is located) or media streams captured by other conference terminals 10. For example, the first conference terminal 10 can acquire media streams from the second conference terminal 10. Conference terminal 10 is also used to send media streams to other conference terminals 10 or servers. For example, the first conference terminal 10 can directly send media streams to the second conference terminal 10, or it can send media streams to the media server 30, which then sends the media streams to the second conference terminal 10.
[0093] The conference terminal 10 can also generate and directly play video feeds. The video feed can be the director's view of the meeting room where the conference terminal 10 is located. There are no restrictions on the view mode in which the conference terminal 10 displays the director's view. For example, the conference terminal 10 can present one or more objects (e.g., people) in a director's close-up mode, or it can present multiple objects (e.g., people) in a director's even distribution mode.
[0094] As an example, Figure 7 An example of a conference terminal structure is shown. The conference terminal 10 includes a processing / control system, a communication unit, a storage unit, and a display unit.
[0095] The processing / control system includes one or more processors 705, which may include a CPU, application processor (AP), modem processor, GPU, image signal processor (ISP), controller, memory, video codec, DSP, baseband processor, NPU, AI processor, etc. Different processing units may be independent devices or integrated into one or more processors. The controller may be one or more combinations of ASIC or FPGA. The processor 705 can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor 705 may also include a storage unit for storing instructions and data. In some embodiments, the storage unit in the processor 705 is a cache memory. This memory can store instructions or data that the processor 705 has just used or is recurring. If the processor 705 needs to reuse the instruction or data, it can directly retrieve it from the memory. This avoids repeated access, reduces the processor 705's waiting time, and thus improves system efficiency.
[0096] The processing / control system can be packaged into a single processor chip, such as a SoC chip or a SIP chip. In one example, the processing / control system can be a system composed of multiple chips; for instance, the baseband processor can be packaged as a separate chip, or integrated with some or all of the circuitry of the RF processing system into a single chip.
[0097] The processing / control system may also include a multimedia unit for handling multimedia-related operations, such as video encoding / decoding and image processing. For example, the multimedia unit may include a camera component 701 for capturing still images or videos. For instance, the camera component 701 of the conference terminal 10 in this embodiment may include one or more cameras. The camera and conference terminal 10 may be an integrated structure or independent of the conference terminal 10. The camera may be used for panoramic shooting of the environment or for shooting a portion of the environment. Optionally, the camera has a directing control function, which can take close-up shots of the current speaker or intelligently distribute the shots among multiple objects in the environment, and transmit the captured images to the conference terminal 10 for display. Optionally, the conference terminal 10 has a directing control function, for example, it can take close-up shots of the images captured by the camera and display the images in a directing close-up mode, or it can intelligently distribute the images captured by the camera and display the images in a directing distribution mode. Alternatively, the camera component 701 may include one or more cameras. The camera may be a panoramic camera or a directing camera. If the camera assembly 701 includes multiple cameras, these multiple cameras may include at least one panoramic camera and at least one directing camera, the panoramic camera being used to capture a panoramic view of the environment, and the directing camera being used to capture a close-up view of the current speaker or sound source.
[0098] The processing / control system may also include a voice unit for processing voice signals. For example, the voice unit may include a microphone 702, a speaker 703, and an audio module 704. The conference terminal 10 can implement audio functions through the microphone 702, speaker 703, and audio module 704, such as audio playback, recording, and sound signal acquisition. Furthermore, in this embodiment, the microphone 702 can be a microphone array, comprising multiple microphones arranged side-by-side. The electronic device uses microphones at different positions in the microphone array to measure the sound signals emitted by the sound source object. Since the sound signals arrive at each microphone with varying degrees of delay, by combining the positions of each microphone in the microphone array, the angle and distance of the sound source object relative to the electronic device can be obtained, thereby achieving sound source localization.
[0099] The storage unit includes a memory 706, which may be an on-chip memory, i.e., located on the processor 507 chip. In one example, the memory 706 may be an off-chip memory, i.e., located outside the processor 507 chip. The memory 706 may be used to store at least a portion of the corresponding computer program instructions and / or data. In one example, one or more processors 705 execute the computer program instructions stored in the memory 706 to implement the relevant operations in the above method embodiments (such as determining a first correction amount for the tilt angle of a first target person in a first image), thereby realizing the image processing method of this application embodiment. The memory 706 storing the corresponding computer program instructions and / or data may mean that the memory 706 stores all the corresponding computer program instructions and / or data for the processor 705 to execute; or it may mean that the memory 706 stores a portion of the corresponding computer program instructions and / or data, including the computer program instructions and / or data that the processor 705 currently needs to execute. The memory 706 may store different portions of the computer program instructions and / or data multiple times for the processor 705 to execute in order to implement the relevant operations in the above method embodiments.
[0100] The aforementioned memory 706 may include one or more of the following storage media: random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), phase-change memory (PCM), resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), cache, register, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), hard disk, etc. In one example, computer program instructions for executing the above embodiments may be stored on non-volatile memory, such as at least a portion of the aforementioned memory 506 (e.g., one or more of ROM, flash memory, EPROM, or hard disk). When the terminal is running, the corresponding computer program instructions may be partially or wholly loaded onto a memory with a faster transfer speed than the processor, such as at least a portion of the memory 506 described above (e.g., one or more of RAM, SRAM, DRAM, PCM, RERAM, MRAM, FRAM, cache, or register), for the processor 507 to execute in order to implement the steps in the above method embodiments.
[0101] The communication unit serves as a communication interface for communication with other components. For example, the communication unit includes a communication module 707, which may include an antenna, a radio frequency (RF) processing unit, etc. The RF processing unit receives RF signals through the antenna and sends the processed signals to the processing / control system for further processing. In the uplink or sidelink direction, the processing / control system processes the terminal-side information and sends it to the RF processing unit, which then processes the signal and transmits it through the antenna.
[0102] In one example, the radio frequency (RF) processing unit serves as the communication interface for external communication of the conference terminal 10, and may include an RF front end (RFFE) and an RF transceiver. The RFFE is primarily used for one or more processing operations, such as shaping, passband selection, or gain adjustment, on the RF signals received by the antenna or those to be transmitted through the antenna. It may include one or more components such as RF switches, duplexers, filters, power amplifiers, antenna tuning, and low-noise amplifiers. The RFFE can be a circuit system composed of multiple discrete devices, or it can be integrated into one or more chips. The RF transceiver processes the RF signals received by the RFFE into baseband / IF signals for further processing by the processing / control system, and processes the baseband / IF signals provided by the processing / control system into RF signals for transmission to the RFFE. The baseband / IF signals transmitted between the RF transceiver and the processing / control system can be digital or analog signals. The RF transceiver can be implemented by one or more chips, which are typically referred to as RF ICs.
[0103] In one example, the RF transceiver and RF front-end can also be packaged on a single chip. In another example, the RF transceiver, RF front-end, and baseband processor can also be packaged on a single chip.
[0104] The display unit is used to display the application's interface, etc. For example, the display unit includes a display screen 708, which includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the conference terminal 10 may include one or N displays 708, where N is a positive integer greater than 1.
[0105] Understandable, Figure 7 The components shown do not constitute a specific limitation on the electronic device. The electronic device in the embodiments of this application may include, but is not limited to, components that are... Figure 7 More or fewer parts. Furthermore, Figure 7The combination / connection relationships between the components can also be adjusted and modified.
[0106] Users in different regions can conduct video conferences through conference terminal 10, thus enabling face-to-face communication. As mentioned above. Figure 6 As shown, users in meeting room #1 can communicate face-to-face with users in meeting room #2 through the first conference terminal in meeting room #1 and the second conference terminal in meeting room #2.
[0107] Considering that in a video conferencing environment, the installation position and angle of the camera component 701 (e.g., a webcam or camcorder) are usually fixed, participants may experience a "top-down" view in the video, resulting in a poor video conferencing experience. In this embodiment, the conferencing terminal 10 or the camera component 701 can correct the perspective of the target object in the video frame, ensuring that the corrected perspective is at eye level or directly facing the viewpoint, thereby enabling eye-level communication and improving the video conferencing experience.
[0108] Using the first and second conference terminals as examples, this section introduces the video conferencing process and the functions of each terminal. Figure 8 The system architecture shown includes a first conference terminal and a second conference terminal. The first and second conference terminals can communicate wirelessly. The first and second conference terminals can have the same structure. The following description uses the first conference terminal as an example to illustrate the specific structure and functions of the conference terminal in this embodiment. In one example, Figure 8 Any functional unit in it can be one or more integrated circuits configured to implement the above methods. Figure 8 The division of units is merely a logical functional division; one function can correspond to one functional unit, or two or more functions can be integrated into one functional unit. In actual implementation, all or some units can be integrated into one physical entity, or distributed across different physical entities. Furthermore, the aforementioned functional units can be implemented in hardware, software, or a combination of both. Whether a function is executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of this application.
[0109] For example, the first conference terminal may include a sound playback module (e.g., a speaker), a sound acquisition device (e.g., a microphone), a video acquisition device (e.g., a camera assembly), a display device (e.g., a display screen), a processor (e.g., a CPU), and a communication unit (e.g., an antenna, a radio frequency front-end, and a radio frequency receiver). Furthermore, the first conference terminal also includes an audio module and an image / video unit. Optionally, the audio module and the image / video module can be implemented in software and integrated into the processor. The image / video module can also be replaced with an image module or a video module, possessing the function of processing images and videos.
[0110] The audio module is used to perform one or more of the following processes on audio data: encoding / decoding, sound source localization, speech detection, etc. Accordingly, the audio module includes modules corresponding to various processes. For example, the audio module may include a sound acquisition unit and a sound source localization unit. The sound acquisition unit acquires the sound signal collected by the sound acquisition device, and the sound source localization unit performs sound source localization on the sound source object emitting the sound signal to determine the reference position of the sound source object relative to the first conference terminal.
[0111] The video module is used to acquire video data from the video capture device during a video conference and process the video data. Video data processing includes at least one or more of the following: encoding / decoding, object detection, object tracking, viewpoint correction, and portrait close-ups. Accordingly, the video module includes various processing modules, such as an image acquisition unit, an object detection unit, an object tracking unit, a viewpoint correction unit, and a director's view control unit. Specifically, the image acquisition unit acquires target images of the environment captured by the video capture device; the object detection unit detects target objects in the target images and extracts their features; the object tracking unit tracks target objects, for example, by determining the position of a reference position obtained based on sound signals in the target image; the viewpoint correction unit corrects the viewpoint of the target objects in the target image; and the director's view control unit performs director's control to determine which director's view is used to output the image / video. The video module also includes an encoding unit and a decoding unit. Figure 8 (Not shown in the image). The encoding unit is used to encode the images acquired by the image acquisition device; the decoding unit is used to decode the received images.
[0112] The processor can be used to implement the functions of the audio module and the image / video module, or it can control the audio module and the image / video module to achieve certain functions. For example, the processor can locate the sound source object and perform directing operations on the sound source object.
[0113] The display device can be used to display video images, such as video images captured during the conference broadcast.
[0114] During the broadcast control process, the audio module acquires audio signals generated from ambient sound signals, and the video module acquires video signals captured from the environment, and performs perspective correction on target objects in the acquired target images. The communication unit transmits the audio and video signals to the second conference terminal.
[0115] The image processing method provided in this application embodiment will be described in detail below with reference to specific scenario drawings.
[0116] The image processing method provided in this application is applicable to electronic devices with image processing capabilities, such as cameras or terminals with cameras, including mobile phones, tablets, wearable devices (e.g., watches, bracelets, helmets, headphones, necklaces, etc.), in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices. This application does not impose any limitations on the specific type of electronic device. Optionally, the electronic device can be a separate electronic device or an integrated electronic device. In a separate electronic device, the terminal host is separate from other devices, which may include at least one of the following: a video input device (e.g., a camera), a video output device (e.g., a display screen), an audio input device (e.g., a microphone (MIC)), and an audio output device (e.g., a speaker). An integrated electronic device means that the electronic device includes a central processing unit (CPU), a video input device, a video output device, an audio input device, and an audio output device.
[0117] In the following description, the method is described using an example of execution by a first electronic device. This first electronic device may be, for example, the aforementioned conference terminal, mobile phone, camera, etc. The first electronic device is deployed in conference room #1. The last user in conference room #1 (e.g., user A) conducts a video conference with at least one user in conference room #2 (e.g., user B) via the first electronic device using a second conference terminal. The steps executed by the first electronic device can also be executed by circuits or chips within the first electronic device responsible for communication functions (such as a modem chip (also known as a baseband chip), or a SoC chip or SIP chip containing a modem core), a chip system, or a processor) and / or circuits or chips responsible for computing functions (such as a CPU, GPU, NPU, AI processor, ASIC, or FPGA); or they can be implemented by logical nodes (or logical modules) and / or software that implement all or part of the functions of the first electronic device. In this embodiment, the processing performed by a single executing entity can also be divided into multiple executing entities, which can be logically and / or physically separated. For ease of description, the first electronic device is described below as a conference terminal. For example, the first electronic device is... Figure 7 The conference terminal 10 shown or Figure 8 The first conference terminal is shown. Furthermore, the following text uses a webcam as an example to illustrate this.
[0118] The following uses the aforementioned application scenario 1 as an example to illustrate the image processing method provided in this application.
[0119] Example 1:
[0120] Please see Figure 9 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The method includes the following steps:
[0121] S901: The first conference terminal acquires a first image, which includes at least one target person.
[0122] The first conference terminal has a conference app installed. Upon detecting the app launch operation, the first conference terminal launches the app, thereby initiating a video conference with the user on the second conference terminal. Optionally, detecting the app launch operation can trigger the camera to start. Once launched, the camera can capture images of the environment. The camera can capture a panoramic image within its actual field of view. The first image can be the panoramic view or a portion of the panoramic view; for example, the first image can be a close-up of an object within the panoramic view. The camera can be installed on the first conference terminal or can be independent of it. The camera can be a webcam that captures images of the environment, or it can be a video camera that captures images of the environment.
[0123] In a video conferencing scenario, the first image may include at least one target person, or the first image may include a facial image of at least one target person. For example, if the current view mode of the first conferencing terminal is the director's close-up mode, based on sound source localization technology, the first conferencing terminal can determine the current speaker in the conference environment, perform director operations on the current speaker, and the first image is a close-up image of the current speaker. If the current view mode of the first conferencing terminal is the director's even distribution mode, the first image acquired by the first conferencing terminal may include multiple target people, and the first image is a stitched image of close-ups of multiple target people.
[0124] S902: The first conference terminal determines whether the viewpoint of the first target person in the first image needs to be adjusted. If so, it executes the process of branch 1; if not, it executes the process of branch 2.
[0125] The first target person belongs to at least one target person. If the first image is a close-up image in director's mode, the first target person is the speaker in the close-up; if the first communication is an image in director's even-sharing mode, the first target person can be any one of multiple target persons. When the first image includes multiple target persons, the image processing flow for each target person can be the same, as illustrated below using the first target person as an example.
[0126] At least one of the target individuals may have one or more facial features where the eyes are not looking directly at the camera. Take the first target individual as an example. The fact that the first target individual's eyes are not looking directly at the camera causes their body to have a tilt angle relative to the plane perpendicular to the camera's viewpoint; in other words, the first target individual has a tilt angle, which is the angle between a first plane and a second plane, where the second plane is perpendicular to the camera's viewpoint. For example, please see... Figure 10 ,like Figure 10 As shown in (a), the camera is shooting at the user from above, causing the first target person in the image (e.g., user A) to have a tilted angle relative to the plane perpendicular to the camera's viewpoint. Figure 10 As shown in (b), plane #A is the first plane, plane #B is the second plane, and α is the angle between the first plane and the second plane (the first angle in this paper), which is less than 90°.
[0127] If the first target person in the first image has a tilt angle or a large first angle (e.g., the first angle is greater than or equal to a first threshold), the first conference terminal outputs the first image to the second conference terminal. Users B and A on the second conference terminal will not be able to communicate in a direct / eye-level manner, resulting in a poor video conferencing experience. To improve the video conferencing experience, when the first angle is greater than or equal to the first threshold, it can be adjusted so that the first target person's perspective is directly facing the camera. Of course, if the first target person in the first image has no tilt angle, or if the tilt angle of the first target person in the first image is less than or equal to the first threshold, the first conference terminal does not need to adjust the tilt angle of the first target person in the first image. The first threshold can be predefined and stored in the first conference terminal.
[0128] The first conference terminal determines whether the viewing angle of the first target person in the first image needs adjustment, that is, whether the first included angle in the first image is greater than or equal to a first threshold. The specific methods used by the first conference terminal to determine whether the viewing angle of the first target person in the first image needs adjustment are not limited.
[0129] For example, in some embodiments, the first conference terminal may calculate the first angle based on one or more of the following parameters: the height of the camera above the ground, the angle between the camera's viewing direction and the horizontal plane (also known as the camera's pitch angle), the horizontal distance from the first target person to the camera, or the vertical distance from the first target person to the camera. The horizontal distance from the first target person to the camera...
[0130] In this embodiment, the camera's installation position is fixed, therefore the camera's height above the ground and the angle between the camera's viewing angle and the horizontal plane are also fixed. In some embodiments, the camera's height above the ground and / or the angle between the camera's viewing angle and the horizontal plane can be stored in a first conference terminal, or the first conference terminal can obtain the camera's height above the ground and / or the angle between the camera's viewing angle and the horizontal plane from other devices, such as a server that stores the camera's height above the ground and / or the angle between the camera's viewing angle and the horizontal plane.
[0131] The horizontal distance from the first target person to the camera is essentially the actual straight-line distance in the physical space, i.e., three-dimensional depth, rather than the depth information directly presented in the first image. It should be understood that the first image is two-dimensional (including X-axis and Y-axis information), and the horizontal distance from the first target person to the camera is actually the distance along the Z-axis. The horizontal distance from the first target person to the camera can be the horizontal distance between the mapped position of the first reference point in the first image in the first space and the camera (which can be simply referred to as the horizontal distance between the first reference point and the camera). It should be understood that the first space is the space where at least one target person is located. The first reference point can be the center position of the first target person. The distance from the first target person to the camera can be calculated using depth estimation techniques; details of depth estimation can be found in the aforementioned related content and will not be repeated here. Alternatively, the horizontal distance from the first target person to the camera can also be calculated using AI methods. For example, the first conference terminal can use / call a preset AI model to calculate the horizontal distance from the first target person to the camera. The AI model can deploy a neural network algorithm. The first conference terminal can use the first image as input to the AI model, which runs the neural network algorithm to obtain an output result, which is used to characterize the inference result of the horizontal distance from the first target person to the camera.
[0132] The vertical distance from the first person to the camera refers to the actual straight-line distance between the first person and the camera in the physical space. For example, the vertical distance from the first person to the camera is the vertical distance between the mapped position of the second reference point in the first image in the first space and the camera (which can be simply referred to as the vertical distance between the first reference point and the camera). The second reference point can be, for example, the center position of the first person, or the head position of the first person. The vertical distance from the first person to the camera can also be calculated based on depth estimation techniques or obtained through AI methods, which will not be elaborated here.
[0133] In possible implementations, one or more of the above parameters have a corresponding / mapping relationship with the included angle. The first conference terminal can determine the first included angle based on one or more of the above parameters and the corresponding relationship. For example, please refer to Table 1, which shows the correspondence between the camera's pitch angle, the horizontal distance between the first reference point and the camera, and the included angle. It should be noted that the value of the first included angle in Table 1 is only an example. According to Table 1, if the camera's pitch angle is 15° and the horizontal distance from the first reference point to the camera is 3 meters, then the first included angle is 5°.
[0134] Table 1
[0135]
[0136]
[0137] In other embodiments, the first conference terminal can use / call a preset AI model to determine whether the first included angle is greater than or equal to a first threshold. The AI model can deploy a neural network algorithm. The first conference terminal can use a first image as input to the AI model, which runs the neural network algorithm to obtain an output result. This output result is used to characterize whether the first included angle needs adjustment. For example, if the output result is "yes," then the first included angle in the first image needs adjustment; if the output result is "no," then the first included angle does not need adjustment.
[0138] As can be seen from Table 1, when the horizontal distance from the first reference point to the camera is large, the first included angle may be 0°. In this case, there is no need to adjust the first included angle, meaning the viewing angle of the first target person in the first image does not need to be adjusted. From this perspective, S902 can be replaced by: the first conference terminal determining whether the horizontal distance from the first reference point to the camera is greater than a preset threshold #1. This preset threshold #1 can be stored in the first conference terminal or predefined. In this case, the first conference terminal does not need to obtain other parameters, such as the camera's pitch angle, the camera's height above the horizontal plane, and the vertical distance from the second reference point to the camera. It also does not need to comprehensively consider the other parameters to determine whether the viewing angle of the first target person in the first image needs adjustment, thereby reducing the processing complexity of the first conference terminal and its power consumption.
[0139] Similarly, as shown in Table 1, when the horizontal distance from the first reference point to the camera is large and the camera's pitch angle is large, the first included angle may be 0°. In this case, there is no need to adjust the first included angle, meaning the viewpoint of the first target person in the first image does not need to be adjusted. From this perspective, S902 can be replaced by: the first conference terminal determining whether the horizontal distance from the first reference point to the camera is greater than a preset threshold #1, and determining whether the camera's pitch angle is greater than a preset threshold #2. Here, the preset threshold #1 can be stored in the first conference terminal or predefined. The preset threshold #2 can also be stored in the first conference terminal or predefined. This approach can also reduce the processing complexity of the first conference terminal and its power consumption.
[0140] Optionally, in a video conferencing scenario, when the first conferencing terminal is in director mode (e.g., director close-up mode), it determines whether the perspective of the first target person in the first image needs adjustment. If the first conferencing terminal's view mode is not director mode, then it is not necessary to determine whether the perspective of the first target person in the first image needs adjustment. In a live streaming scenario, when the first conferencing terminal is in live streaming mode, it determines whether the perspective of the first target person in the first image needs adjustment; if the first conferencing terminal is not in live streaming mode, then it is not necessary to determine whether the perspective of the first target person in the first image needs adjustment. The first conferencing terminal determines whether the perspective of the first target person in the first image needs adjustment only based on the view mode, which has lower processing complexity and can reduce the power consumption of the first conferencing terminal.
[0141] Optionally, S902 is not a mandatory step. For example, it can be predefined whether the viewpoint of the first target person in the first image needs to be adjusted. Or, for example, it can be predefined whether the viewpoint of the first target person in the first image needs to be adjusted in either the director's close-up mode or the director's even-distribution mode. From this perspective, S902 can be replaced by: the first conference terminal determining whether the current view mode is director's close-up mode or director's even-distribution mode.
[0142] For branch 2, there is no need to adjust the perspective of the first target person in the first image; the first image can be output to the second conference terminal. The following mainly describes the process of branch 1, which includes determining the first correction amount of the first included angle and adjusting the first included angle according to the first correction amount.
[0143] S903: The first conference terminal determines the first correction amount of the first included angle.
[0144] Because the initial angle is relatively large, visually, the eyes of the target person are not directly facing the camera. Therefore, the initial angle can be reduced to ensure that the eyes of the target person are facing the camera as much as possible. The first conference terminal can adjust the initial angle once or multiple times. The initial correction amount can be the amount adjusted once or multiple times.
[0145] Generally, the higher the camera is above the ground, the larger the first angle; the larger the camera's pitch angle, the larger the first angle; the greater the horizontal distance from the target object in the image to the camera, the larger the first angle. Therefore, the first correction amount can be determined based on one or more of the following: the camera's height above the ground, the camera's pitch angle, the horizontal distance from the target object in the image to the camera, and the vertical distance from the target object to the camera.
[0146] The horizontal distance from each target person to the camera varies depending on the number of people included in the image. For ease of description, this document refers to the horizontal distance from the target person to the camera in the image used to determine the first correction amount as the horizontal distance between the mapped position of the first reference point in the first image and the camera in the first space. If at least one target person is the first target person, then the first reference point can be the center position of the first target person. If at least one target person includes multiple target persons, then the first reference point is determined based on the horizontal distance between one or more of these multiple target persons and the camera. For example, if at least one target person includes a first target person and a second target person, then the first reference point is determined based on the horizontal distance between the first target person and the second target person and the camera, respectively.
[0147] Similarly, the vertical distance from the target person to the camera varies depending on the number of target persons included in the image. In this paper, the vertical distance from the target person to the camera in the image used to determine the first correction amount is collectively referred to as the horizontal distance between the mapped position of the second reference point in the first image in the first space and the camera. If at least one target person is the first target person, then the second reference point can be the center position of the first target person, or the head position of the first target person. If at least one target person includes multiple target persons, then the second reference point is determined based on the vertical distance between one or more of these multiple target persons and the camera. For example, if at least one target person includes the first target person and the second target person, then the first reference point is determined based on the vertical distance between the first target person and the second target person and the camera, respectively.
[0148] The first conference terminal can determine a first correction amount based on a first correspondence and at least one parameter. The first correspondence includes (or is) a correspondence between at least one parameter and a correction angle. The at least one parameter may be the height of the camera above the ground, the camera's pitch angle, the horizontal distance from the first reference point to the camera, and the vertical distance from the second reference point to the camera. The first correspondence may be stored in the first conference terminal, or the first conference terminal may obtain the first correspondence from other devices (e.g., a server). For example, please refer to Table 2, which illustrates the first correspondence. Table 2 uses an example where at least one parameter includes the camera's pitch angle and the horizontal distance from the first reference point to the camera, and the first correction amount in Table 2 is merely an example.
[0149] Table 2
[0150]
[0151] The first conference terminal can determine the camera's tilt angle and the horizontal distance from the first reference point to the camera. According to Table 1, the camera's tilt angle is 15°, and the horizontal distance from the first reference point to the camera is 3 meters. Therefore, the first correction is 5°. For details on how the first conference terminal determines the camera's height from the ground, the camera's tilt angle, the horizontal distance from the first reference point to the camera, and the vertical distance from the target person to the camera, please refer to the relevant content in S902 above; it will not be repeated here.
[0152] It should be noted that Tables 1 and 2 use the example where the first correction amount and the first included angle are the same. If the first correction amount and the first included angle are the same, the first included angle can be reduced based on the first correction amount, for example, the first included angle can be adjusted to a second included angle, where the second included angle is 0°. In some embodiments, the first correction amount can be less than the first included angle.
[0153] Furthermore, as mentioned above, if the first included angle is small, there is no need to adjust the first included angle, and naturally there is no need to determine the first correction amount. Therefore, if the first included angle is greater than or equal to the first threshold, S903 is executed; or, if the first reference point and the camera are less than or equal to a preset threshold #3, the first correction amount is determined to minimize unnecessary calculation operations and save power consumption of the first conference terminal.
[0154] S904: The first conference terminal adjusts the first included angle to the second included angle according to the first correction amount, and obtains and outputs the second image.
[0155] After determining the first correction amount, the first conference terminal can adjust the first included angle to a second included angle based on the first correction amount to adjust the line of sight of the first target person. The image formed after adjusting the first included angle is called the second image. In the second image, the eyes of the first target person can look directly at the camera. The first conference terminal outputs the second image, for example, and sends it to the second conference terminal. The user of the second conference terminal can see the frontal image of the first target person, thereby improving the video conferencing experience.
[0156] As an example, the first conference terminal can adjust the first included angle to the second included angle in a single adjustment. For instance, if the first image is a frame in director's close-up mode or director's even-shading mode, then the first conference terminal can adjust the first included angle to the second included angle in a single adjustment. The first conference terminal rotates the first included angle by a first correction amount, thus changing the first included angle to the second included angle. Figure 11 As shown, Figure 11 (a) in the image is the first image. Figure 11 In (b), α is the first included angle. Figure 11 In (c), β is the second included angle. Figure 11 (d) in the image is the second image. Figure 11 (a) to Figure 11The person in (d) is the primary target. From... Figure 11 It can be seen that by adjusting the first angle to the second angle, the line of sight of the first target in the second image can be made closer to looking directly at the camera. It should be noted that... Figure 11 For ease of illustration, we'll use an example where β is less than α and not 0. In possible implementations, β can be 0, meaning the first and second planes are parallel, so that the eyes of the first target in the second image are directly facing the camera.
[0157] As an example, the first conference terminal can adjust the first included angle to the second included angle through multiple adjustments. It should be understood that in a video conferencing scenario, both image input and output are real-time. In other words, the images captured by the camera are also changing in real time, and the image output by the camera can be all or part of the images captured by the camera at the same moment. For example, the image output by the camera at the k-th frame after the current frame is the image or part of the image input by the camera at the k-th frame after the current frame, not the image or part of the image input by the camera in the current frame. Furthermore, in possible scenarios, during smooth transitions, when switching from one view mode (e.g., the characteristic view mode) to another view mode (e.g., the evenly distributed view mode), the screen does not switch directly between the two frames. There is a transition frame between the two frames before and after the switch, and the transition frame outputs the intermediate image between the screen before and after the switch to ensure a smooth transition.
[0158] During the switch of the director's view, the first conference terminal can adjust the first included angle to the second included angle through at least two adjustments, and the sum of the adjustment angles of the multiple adjustments is the first correction amount. For example... Figure 12 As shown, Figure 12 Take, for example, the first included angle being adjusted to the second included angle after three adjustments. Figure 12 (A) in the image is the first image. Figure 12 (D) in the image is the second image. Figure 12 (B) and (C) are transitional image frames where the first image is adjusted to the second image. (a) to (d), corresponding one-to-one with (A) to (D), show the viewpoint of the first target person. From Figure 12 It can be seen that by adjusting the first included angle (α#1) to α#2, α#3, and the second included angle (0°) in sequence, as follows... Figure 12 In (a) to (d), the first image changes from (A) to (D) (i.e., the second image). It can be seen that in the second image, the eyes of the first target person are facing the camera.
[0159] Optionally, the first included angle is adjusted to the second included angle through at least two adjustments based on at least one step size and a first correction amount. The step sizes for the different adjustments may be the same or different.
[0160] In the equal distribution mode of the broadcast director, for multiple target people, the first conference terminal can adjust the eye gaze of other target people in the same way as the eye gaze of the first target person, and then stitch the images of the target people after the adjustment of the viewpoint to generate a second image.
[0161] In Embodiment 1, if the eyes of the first target person in the first image are not directly facing the camera, the first conference terminal can adjust the first angle so that the eyes of the first target person are directly facing the camera after the angle is adjusted, thereby improving the conference video experience.
[0162] In video conferencing scenarios, new participants may join, and participants may leave, which can affect the display of target individuals in the image. Therefore, in this embodiment, the first conferencing terminal can adaptively adjust the first angle corresponding to each target individual to ensure that each target individual's line of sight is as close as possible to the camera. Please refer to the description in Embodiment 2 for details. Embodiment 2 is applicable to scenarios with automatic framing.
[0163] Example 2:
[0164] Please see Figure 13 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The method includes the following steps:
[0165] S1301: The first conference terminal acquires a first image, which includes multiple target figures.
[0166] The difference between S1301 and S901 is that the first image can be an image from the automatic viewfinder mode. For details on S1301, please refer to the relevant content in S901; it will not be repeated here.
[0167] S1302: The first conference terminal determines the correction amount of the perspective of multiple target persons.
[0168] When there are multiple target figures in the first image, the viewing angles of each target figure can be adjusted separately. Taking one target figure as an example, the first conference terminal can determine the correction amount of the viewing angle based on a first correspondence and at least one parameter. The at least one parameter includes one or more of the following: the height of the camera from the horizontal plane, the camera's pitch angle, the horizontal distance between the first reference point and the camera, and the vertical distance between the second reference point and the camera. The first correspondence is described in the relevant content of Embodiment 1.
[0169] For a specific target person (e.g., the first target person), the first reference point can be determined based on the horizontal distances between multiple target persons and the camera. For example, the first reference point is a position with a first horizontal distance from the camera, which is the average or weighted average of the horizontal distances between multiple target persons and the camera.
[0170] Similarly, the second reference point can be determined based on the vertical distances between multiple target individuals and the camera. For example, the first reference point is a position where the vertical distance between the target individuals and the camera is the second distance, which is the average or weighted average of the vertical distances between multiple target individuals and the camera.
[0171] For each target person, the first conference terminal can calculate the correction amount corresponding to the perspective of each target person, and obtain multiple correction amounts.
[0172] S1303: The first conference terminal adjusts the perspective of multiple target individuals according to the determined multiple correction values.
[0173] The first conference terminal determines multiple correction values and adjusts the viewing angles of multiple target individuals based on these values, with each correction value corresponding to one target individual. For example, the first correction value corresponds to the first target individual. Before correction, the tilt angle of the first target individual is the first included angle; based on the first correction value, the first included angle is adjusted to the second included angle. For the specific correction process for each target individual, please refer to the relevant content in S904 above.
[0174] For example, such as Figure 14 As shown, Figure 14 (a) and (b) in the example take a multi-target person as two target persons (e.g., the first target person and the second target person). Figure 14 (a) in the image is the first image. Figure 14 (b) in the image is the second image. From the first image to the second image, the correction amount of the perspective of the first target person is 10°, and the correction amount of the perspective of the second target person is 10°. Figure 14 Examples (A) and (B) in the text use multiple target individuals as the primary target. Figure 14 (A) in the image is the first image. Figure 14 Image (B) is the second image. The correction amount for the perspective of the first target person from the first image to the second image is 20°. Figure 14 It can be seen that when there are multiple target individuals, adjusting the perspective of each target individual can make it possible for all target individuals to look directly at the camera.
[0175] S1304: The second target person is detected to have moved out of the camera's shooting range. Determine whether the horizontal distance between the second target person and the camera is greater than or equal to a preset threshold #1. If it is less than or equal to, execute branch 1; if it is greater than, execute branch 2.
[0176] During a video conference, a participant (the second target person) may leave the camera's field of view. This departure may affect the perspective of other target persons in the subsequent video feed. Therefore, when a participant leaves, the perspective correction amount for one or more target persons can be re-determined. For ease of description, we will use the example of a second target person leaving.
[0177] If the horizontal distance between the second target person and the camera is relatively far, it will have little or no impact on the viewing angles of other target persons in the subsequent video frame. Therefore, when the second target person is detected to have moved out of the camera's shooting range, it can be determined whether the horizontal distance between the second target person and the camera is greater than or equal to a preset threshold #1. If it is less than or equal to, then branch 1 is executed, that is, the viewing angle correction amount corresponding to one or more target persons is re-determined. If it is greater than, branch 2 can be executed, for example, the current image can be directly output without re-determining the viewing angle correction amount corresponding to one or more target persons, so as to reduce the power consumption of the first conference terminal.
[0178] S1305: Redetermine the first correction amount corresponding to the first target person.
[0179] Branch 1 takes the re-determination of the first correction value corresponding to the first target person as an example. When the second target person is detected moving out of the camera's shooting range, and the horizontal distance from the second target person to the camera is determined to be less than or equal to a preset threshold #1, the first correction value corresponding to the first target person is re-determined. For the specific implementation of the first conference terminal re-determining the first correction value corresponding to the first target person, please refer to the relevant content in S903, which will not be elaborated here.
[0180] S1306: Adjust the first included angle to the second included angle according to the first correction amount.
[0181] For details regarding S1306, please refer to the relevant content in S904 mentioned above; it will not be repeated here.
[0182] Figure 13Let's take the example of a second target person leaving the meeting midway. It should be understood that during a video conference, new participants (such as a third target person) may join the camera's field of view. The addition of a third target person may affect the perspective of other target persons in the subsequent video frame. Therefore, when a new participant joins, the perspective correction amount for one or more target persons can be re-determined. For ease of description, let's take the newly joined participant as the third target person as an example.
[0183] If the horizontal distance between the third target person and the camera is relatively far, it will have little or no impact on the viewing angles of other target persons in the subsequent video frame. Therefore, when the third target person is detected to have moved out of the camera's shooting range, it can be determined whether the horizontal distance between the third target person and the camera is greater than or equal to a preset threshold #1. If it is less than or equal to, then branch 1 is executed, that is, the viewing angle correction amount corresponding to one or more target persons is re-determined. If it is greater than, branch 2 can be executed, for example, the current image can be directly output without re-determining the viewing angle correction amount corresponding to one or more target persons, thereby reducing the power consumption of the first conference terminal.
[0184] In Example 2, if a new participant joins or a participant leaves the meeting, and the horizontal distance between the new or departing participant and the camera is relatively close, the adjustment amount of the remaining participants' viewing angles is recalculated to ensure that their eyes are directly facing the camera, thus improving the video conferencing experience. If a new participant joins or a participant leaves the meeting, and the horizontal distance between the new or departing participant and the camera is relatively large, the current image frame is output to minimize meeting latency and further enhance the video conferencing experience.
[0185] In the embodiments provided above, the method provided by the embodiments of this application is described using the execution of a first conference terminal as an example. In this application, each embodiment can be implemented independently or in combination based on certain inherent connections; in each embodiment, different implementation methods can be implemented in combination or independently. To achieve the functions in the methods provided in the embodiments of this application above, the first conference terminal may include a hardware structure and / or software modules, implementing the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. Whether a particular function is executed in the form of a hardware structure, software module, or a hardware structure plus software modules depends on the specific application and design constraints of the technical solution.
[0186] Based on the same inventive concept as the method embodiments, this application provides a communication device. The communication device used to implement the above method in the embodiments of this application is described below with reference to the accompanying drawings. The content above can be used in subsequent embodiments, and repeated content will not be repeated.
[0187] Figure 15 This is a schematic block diagram of a communication device 1500 provided in an embodiment of this application. The communication device 1500 can correspondingly implement the functions or steps implemented by the first conference terminal in the various method embodiments described above. The communication device 1500 may include modules or units that implement the methods described above; for example, the communication device 1500 includes an acquisition unit 1510, a determination unit 1520, and an adjustment unit 1530. Optionally, the communication device may further include a storage unit, which can be used to store instructions (code or program) and / or data. The storage unit may be, for example, a memory. The acquisition unit 1510, the determination unit 1520, and the adjustment unit 1530 may be coupled to the storage unit. For example, one or more of the acquisition unit 1510, the determination unit 1520, and the adjustment unit 1530 can read instructions (code or program) and / or data from the storage unit to implement the corresponding method.
[0188] The acquisition unit 1510, determination unit 1520, and adjustment unit 1530 can be one or more functional units divided by a processor. The processor can be, for example, a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, a programmable logic device (PLD), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc. When the communication device 1500 is a system-on-a-chip, the processor can be an FPGA, a dedicated ASIC, a SoC, a CPU, a network processor (NP), a DSP, an MCU, a PLD, or other integrated chips. The processor can be a processor that includes a system-on-a-chip.
[0189] Optionally, the communication device 1500 further includes a communication unit, which may be a transceiver, interface circuit, bus, pin, or other possible communication interface for receiving signals from other devices. For example, when the device is implemented as a chip, the communication unit may be an interface circuit for the chip to receive signals from other chips or devices, or an interface circuit for the chip to send signals to other chips or devices. For example, the interface circuit may be a code / data read / write interface circuit. The interface circuit may be used to receive code instructions (which are stored in memory and can be read directly from memory or via other devices) and transmit them to one or more of the acquisition unit 1510, determination unit 1520, and adjustment unit 1530. As another example, the interface circuit may also be a signal transmission interface circuit between a communication processor and a transceiver. When the communication device 1500 is a chip-type device or circuit, the communication unit may be an input / output circuit and / or a communication interface; the acquisition unit 1510, determination unit 1520, or adjustment unit 1530 may be an integrated processor, microprocessor, or integrated circuit.
[0190] When the communication device 1500 is a module or unit within a device, the storage unit can be a storage unit within the chip, such as a register or cache. For example, the storage unit can also be a storage unit located outside the chip within a terminal device or access network device, such as read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM), flash memory, or hard disk. The aforementioned units can be configured independently or partially or completely integrated. In one example, computer program instructions for executing the above embodiments can be stored in non-volatile memory, such as at least a portion of the aforementioned memory (e.g., one or more of ROM, flash memory, or hard disk). When the terminal device or access network device is running, the corresponding computer program instructions may be partially or wholly loaded into one or more of the faster memory among the acquisition unit 1510, determination unit 1520 and adjustment unit 1530, such as at least a portion of the memory (e.g., one or more of RAM, cache, or register), for execution by one or more of the acquisition unit 1510, determination unit 1520 and adjustment unit 1530 to implement the steps in the above method embodiments.
[0191] In one implementation, the communication device 1500 can correspondingly implement the behavior and functions of the first conference terminal in the above method embodiments. The communication device 1500 can be a terminal device (e.g., a mobile phone, a laptop computer), a component within the terminal device (e.g., a module, chip, or circuit), a part of a module, chip, or chipset in the terminal device used to execute the relevant method functions, or a software module in the terminal device capable of implementing the above communication method; there are no limitations. For details, please refer to the relevant content of the foregoing method embodiments, which will not be repeated here.
[0192] For example, the acquisition unit 1510 is used to acquire a first image captured by the camera component, the first image including at least one target person, the at least one target person including a first target person. The determination unit 1520 is used to determine a first correction amount for the tilt angle of the first target person, the tilt angle of the first target person being a first angle between a first plane and a second plane where the first target person is located, the second plane being a plane perpendicular to the viewing angle of the camera component. The adjustment unit 1530 is used to adjust the first angle to a second angle according to the first correction amount, obtain and output a second image, the second angle being smaller than the first angle.
[0193] As an optional design, the determining unit 1520 is specifically used to: acquire at least one parameter, and determine a first correction amount based on the at least one parameter and the correspondence between the at least one parameter and the correction angle. The at least one parameter includes one or more of the following: the horizontal distance between the mapped position of a first reference point in the first image in the first space and the camera component; the vertical distance between the mapped position of a second reference point in the first image in the first space and the camera component; the height of the camera component above the ground; or the angle between the viewing direction of the camera component and the horizontal plane. The first space is the space where at least one target person is located.
[0194] As an optional design, the determining unit 1520 is specifically used to: determine a first correction amount when the distance between the first reference point and the camera component is less than or equal to a preset threshold; or, at least one target person includes a second target person, and the first correction amount is determined when the second target person is detected to have moved out of the shooting range of the camera component and the horizontal distance between the second target person and the camera component is less than or equal to a preset threshold; or, the first correction amount is determined when a third target person is detected to have entered the shooting range of the camera component and the horizontal distance between the third target person and the camera component is less than or equal to a preset threshold.
[0195] As an optional design, at least one target person is designated as the first target person, the first reference point is designated as the center position of the first target person, and the second reference point is designated as the center position of the first target person; or, at least one target person is designated as multiple target persons, the first reference point is determined based on the horizontal distance between the multiple target persons and the camera component, and the second reference point is determined based on the vertical distance between the multiple target persons and the camera component.
[0196] As an optional design, the adjustment unit 1530 is specifically used to: adjust the first included angle to a second included angle through at least two adjustments based on at least one step size and a first correction amount, wherein the sum of the at least two adjustment angles is equal to the first correction amount.
[0197] As an optional design, the first image is an image frame in the director's close-up mode or the director's equal-sharing mode; or, the first image is an image frame during the switching of multiple view modes, including the director's close-up mode and the director's equal-sharing mode.
[0198] This application also provides a computer-readable storage medium for storing a computer program. When the computer program is executed by a computer, the computer can implement the method provided in the above-described method embodiments.
[0199] This application also provides a computer program product for storing a computer program. When the computer program is executed by a computer, the computer can implement the method provided in the above-described method embodiments.
[0200] This application also provides a chip, including a processor coupled to a memory, for calling a program in the memory to enable the chip to implement the method provided in the above-described method embodiments.
[0201] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0202] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0205] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. An image processing method, characterized in that, include: Acquire a first image captured by a camera component, the first image including at least one target person, the at least one target person including the first target person; A first correction amount is determined for the tilt angle of the first target person, wherein the tilt angle of the first target person is the first angle between the first plane and the second plane where the first target person is located, and the second plane is a plane perpendicular to the viewing angle of the camera component. The first included angle is adjusted to the second included angle according to the first correction amount, and a second image is obtained and output, wherein the second included angle is smaller than the first included angle.
2. The method as described in claim 1, characterized in that, Determining the first correction amount for the tilt angle of the first target person includes: Get at least one parameter; The first correction amount is determined based on the at least one parameter and the correspondence between the at least one parameter and the correction angle; Wherein, the at least one parameter includes one or more of the following parameters: The horizontal distance between the mapped position of the first reference point in the first image in the first space and the camera component, wherein the first space is the space where the at least one target person is located; The height of the camera component above the ground; The angle between the viewing angle of the camera component and the horizontal plane; The vertical distance between the mapped position of the second reference point in the first image in the first space and the camera component, wherein the first space is the space where the at least one target person is located.
3. The method as described in claim 2, characterized in that, Determining the first correction amount for the first included angle includes: The first correction amount is determined when the distance between the first reference point and the camera component is less than or equal to a preset threshold; or, The at least one target person further includes a second target person. If the second target person is detected to have moved out of the shooting range of the camera component, and the horizontal distance from the second target person to the camera component is less than or equal to the preset threshold, the first correction amount is determined; or, If a third target person is detected entering the shooting range of the camera component, and the horizontal distance between the third target person and the camera component is less than or equal to the preset threshold, the first correction amount is determined.
4. The method according to any one of claims 1-3, characterized in that, The at least one target person is the first target person, the first reference point is the center position of the first target person, and the second reference point is the center position of the first target person; or... The at least one target person can be multiple target persons. The first reference point is determined based on the horizontal distance between the multiple target persons and the camera component, and the second reference signal is determined based on the vertical distance between the multiple target persons and the camera component.
5. The method according to any one of claims 1-4, characterized in that, Adjusting the first included angle to the second included angle based on the first correction amount includes: The first included angle is adjusted to the second included angle through at least two adjustments based on at least one step size and the first correction amount, wherein the sum of the at least two adjustment angles is equal to the first correction amount.
6. The method according to any one of claims 1-5, characterized in that, The first image is a frame from the director's close-up mode or the director's even distribution mode; or, The first image is an image frame during the switching of multiple view modes, including the director's close-up mode and the director's even distribution mode.
7. An image processing apparatus, characterized in that, include: An acquisition unit is configured to acquire a first image captured by a camera component, the first image including at least one target person, the at least one target person including the first target person; The determining unit is used to determine a first correction amount for the tilt angle of the first target person, wherein the tilt angle of the first target person is the first included angle between the first plane and the second plane where the first target person is located, and the second plane is a plane perpendicular to the viewing angle of the camera component. The adjustment unit is used to adjust the first included angle to a second included angle according to the first correction amount, obtain and output a second image, wherein the second included angle is smaller than the first included angle.
8. The apparatus as claimed in claim 7, characterized in that, The determining unit is specifically used for: Get at least one parameter; The first correction amount is determined based on the at least one parameter and the correspondence between the at least one parameter and the correction angle; Wherein, the at least one parameter includes one or more of the following parameters: The horizontal distance between the mapped position of the first reference point in the first image in the first space and the camera component, wherein the first space is the space where the at least one target person is located; The height of the camera component above the ground; The angle between the viewing angle of the camera component and the horizontal plane; The vertical distance between the mapped position of the second reference point in the first image in the first space and the camera component, wherein the first space is the space where the at least one target person is located.
9. The apparatus as claimed in claim 8, characterized in that, The determining unit is specifically used for: The first correction amount is determined when the distance between the first reference point and the camera component is less than or equal to a preset threshold; or, The at least one target person further includes a second target person. If the second target person is detected to have moved out of the shooting range of the camera component, and the horizontal distance from the second target person to the camera component is less than or equal to the preset threshold, the first correction amount is determined; or, If a third target person is detected entering the shooting range of the camera component, and the horizontal distance between the third target person and the camera component is less than or equal to the preset threshold, the first correction amount is determined.
10. The apparatus according to any one of claims 7-9, characterized in that, The at least one target person is the first target person, the first reference point is the center position of the first target person, and the second reference point is the center position of the first target person; or... The at least one target person can be multiple target persons. The first reference point is determined based on the horizontal distance between the multiple target persons and the camera component, and the second reference signal is determined based on the vertical distance between the multiple target persons and the camera component.
11. The apparatus according to any one of claims 7-10, characterized in that, The adjustment unit is specifically used for: The first included angle is adjusted to the second included angle through at least two adjustments based on at least one step size and the first correction amount, wherein the sum of the at least two adjustment angles is equal to the first correction amount.
12. The apparatus according to any one of claims 7-11, characterized in that, The first image is a frame from the director's close-up mode or the director's even distribution mode; or, The first image is an image frame during the switching of multiple view modes, including the director's close-up mode and the director's even distribution mode.
13. An image processing apparatus, characterized in that, include: One or more processors and a memory, wherein the one or more processors are configured to perform the method as described in any one of claims 1-6, and the memory is used to store data and / or instructions.
14. A readable storage medium, characterized in that, The readable storage medium includes a program that, when run on the device, causes the device to perform the method as described in any one of claims 1-6.
15. A chip system, characterized in that, include: The chip system includes at least one chip and a memory, wherein the at least one chip is used to read and execute a program stored in the memory to implement the method as described in any one of claims 1-6.
16. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.
