Cross-device camera sharing method and electronic device

Abstract

The invention provides a cross-device camera sharing method and an electronic device, and relates to the technical field of shooting. According to the cross-device camera sharing method provided by the invention, when the first electronic device and the second electronic device are in the smart interconnection, and the video application in the second electronic device calls the camera in the first electronic device, a user can shoot the camera in the required shooting mode in the smart interconnection of the first electronic device or the second electronic device; the shooting mode comprises a portrait centering mode and / or a microspur mode, by adopting the method in the embodiment of the invention, after the two devices establish the interconnection service, when the application of the second electronic device calls the camera of the first electronic device to shoot the image, the first electronic device can shoot the image; the first electronic device can provide at least one special effect mode for the shooting application in the second electronic device through the interconnection service so as to meet the shooting requirement of a user.

Description

Technical Field

[0001] The present application relates to the field of photographing technology, and in particular to a method for sharing a camera across devices and an electronic device. Background Art

[0002] With the continuous development of smart terminal devices, they generally support interconnected services. Interconnected services enable two connected devices to share some functions. For example, if both a mobile phone and a computer support interconnected services, when the mobile phone and computer are connected through interconnected services, third-party applications on the computer (such as conferencing applications, live broadcast applications, instant messaging applications, etc.) can use the mobile phone's camera to capture images.

[0003] However, when the third-party application does not set the shooting effects required by the user (such as the effect of centering the portrait or the effect of magnifying a part), it affects the effect of the computer using the camera to capture images and affects the user's experience of using Internet services. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides a method and electronic device for sharing cameras across devices, so that when two devices establish an interconnected service, when the application of the second electronic device calls the camera of the first electronic device to capture an image, the first electronic device can provide at least one special effects mode for the second electronic device through the interconnected service, thereby meeting the user's shooting needs.

[0005] In a first aspect, an embodiment of the present application provides a method for sharing a camera across devices, which is applied to a first electronic device, wherein the first electronic device and the second electronic device each have a first application installed, and the first electronic device establishes a communication connection with the second electronic device through the first application; the method includes: when the second application of the second electronic device calls the camera of the first electronic device, the first electronic device obtains a target shooting instruction, and the target shooting instruction is generated according to the target shooting mode selected by the user through the first application of the first electronic device or through the first application of the second electronic device, and the first application is different from the second application; wherein the shooting mode to be selected includes a first shooting mode and / or a second shooting mode, the first shooting mode is used to indicate that the target object is centered in the target image, and the second shooting mode is used to indicate that the target area of ​​the target object is enlarged and displayed in the target image; an image is shot according to the target shooting mode indicated by the target shooting instruction to obtain a target image; the target image is transmitted to the second electronic device, and the second electronic device displays the target image on the second application interface.

[0006] Exemplarily, the first electronic device can be a mobile phone, a tablet computer, a smart watch, or the like. The second electronic device can be a computer, a mobile phone, or the like. The first electronic device is installed with a camera. In this example, the first electronic device can be exemplified by a mobile phone, and the second electronic device can be exemplified by a computer.

[0007] Exemplarily, the first application can be a smart interconnection application, and the two electronic devices interconnected through the first application can share the camera with each other.

[0008] Exemplarily, the second application is an application supporting a shooting service, such as a live broadcast application, an instant messaging application, or the like.

[0009] Exemplarily, the first shooting mode can be a portrait centering mode, i.e., a face is centered in a shooting image, and the second shooting mode is a macro mode, e.g., a selected organ in a face can be enlarged.

[0010] Exemplarily, the first application can provide the first shooting mode, or the second shooting mode, or the first shooting mode and the second shooting mode.

[0011] Exemplarily, the target object can be a face, an animal, an object, or the like. In this example, the target object is exemplified by a face.

[0012] In this way, the first application in the first electronic device provides at least one shooting mode, so that when the second application of the second electronic device calls the camera of the first electronic device, even if the second application does not provide the first shooting mode or the second shooting mode, the first electronic device can also shoot according to the first shooting mode or the second shooting mode, meeting the user's demand for shooting special effects when using the camera across devices. In addition, the user can not only select the target shooting mode on the first electronic device but also select the target shooting mode on the second electronic device, achieving the purpose of selecting the shooting mode on both ends (i.e., the first electronic device and the second electronic device), and improving the flexibility of user use.

[0013] According to the first aspect, the target image is obtained by shooting an image according to the target shooting mode indicated by the target shooting instruction, including: in response to the target shooting instruction, obtaining an image shot by the first electronic device as an initial image; identifying a target object in the initial image; in a case where the target object is identified, obtaining image processing data of the initial image according to the target shooting mode and the identification result, the image processing data including size information of the initial image and coordinates of key points of the target object; and performing target image processing on the initial image according to the target shooting mode and the image processing data to obtain the target image, the target image processing including cropping processing on the initial image and enlargement processing on the cropped image.

[0014] Exemplarily, the recognition result may include a target object annotation box. For example, when the target object may be a face, the recognition result may include a face annotation box.

[0015] For example, the size information of the initial image may include the height and width of the initial image. The key points of the target object include the center point of the target object (such as the center point of a human face) or feature points in the target object. For example, when the target object is a human face, the feature points may include the center point of the nose, the center point of the eyes, and the center point of the mouth. For example, when the target object is a bottle, the feature points may include the center point of the bottle mouth and the center point of the bottle body.

[0016] In this way, when the first electronic device identifies the target object in the initial image, it captures the image according to the target capture mode, avoiding situations where the captured image does not meet the user's needs due to user error. The first electronic device uses the initial image's size information and the coordinates of the target object's key points to crop the initial image, ensuring that the cropped area is appropriate and does not extend beyond the initial image, resulting in the target object being miscentered. Furthermore, the acquired image processing data facilitates determining the appropriate magnification ratio, avoiding the problem of the magnified image exceeding the display area.

[0017] According to the first aspect, in response to a target shooting instruction, an image captured by a first electronic device is obtained as an initial image, including: in response to the target shooting instruction, an image captured by a camera at a first zoom rate is obtained as the initial image; when the target shooting instruction indicates that the target shooting mode is the first shooting mode, after identifying the target object in the initial image, the method also includes: adjusting the zoom rate of the camera to a second zoom rate corresponding to the ultra-wide-angle mode; obtaining an image captured in the ultra-wide-angle mode; and updating the initial image with the image captured in the ultra-wide-angle mode.

[0018] Exemplarily, the first zoom ratio may be a default zoom ratio for images captured by a camera in the first electronic device, such as a zoom ratio of 1. The second zoom ratio may be a maximum zoom ratio supported by the ultra-wide-angle mode, such as a second zoom ratio of 0.5.

[0019] In this way, after recognizing that there is a target object in the initial image, the camera's zoom rate is adjusted to the zoom rate corresponding to the ultra-wide-angle mode, that is, the image is captured in the ultra-wide-angle mode at this time, so that the field of view of the captured image is widened, thereby making the target object's activity range larger and less likely to go out of the picture.

[0020] According to the first aspect, when a target object is identified, image processing data of an initial image is obtained according to the target shooting mode and the recognition result, including: when the target shooting mode is a first shooting mode, obtaining the size information of the initial image, obtaining the coordinates of a first key point of the target object, and obtaining the width of a target object annotation box in a cropping coordinate system, where the first key point is the center point of the target object; when the target shooting mode is a second shooting mode, obtaining the size information of the initial image, obtaining the coordinates of each key point in the target object, and the width of the target object annotation box in a cropping coordinate system, where the key points of the target object include the center points of each preset area to be magnified in the target object.

[0021] In this way, different shooting modes require different image processing data, and the first electronic device can obtain different image processing data according to the target shooting mode, so as to facilitate the subsequent flexible processing of the initial image.

[0022] According to a first aspect, target image processing is performed on an initial image according to a target shooting mode and target image processing data to obtain a target image, including: when the target shooting mode is a first shooting mode, obtaining a first magnification ratio; determining a reference position of a cropping area in a cropping coordinate system according to the first magnification ratio, the coordinates of a first key point, and size information of the initial image; converting the reference position in the cropping coordinate system into a reference position in a real image coordinate system; using the first key point as the center of the cropping area and the reference position in the real image coordinate system as the reference for cropping, cropping the initial image to obtain a first cropped image; and enlarging the first cropped image according to the first magnification ratio to obtain a target image.

[0023] Illustratively, the cropping coordinate system is a cropping region coordinate system, and the reference position of the cropping region is conveniently determined in the cropping coordinate system.

[0024] Exemplarily, the reference position may be the lower left corner of the cropping area.

[0025] In this way, the first electronic device can quickly determine the reference position of the cropping area using the first magnification ratio, the coordinates of the first key point, and the size information of the initial image. The reference position in the cropping coordinate system is converted to the reference position in the real image coordinate system, facilitating subsequent cropping of the initial image. Furthermore, using the first key point as the center of the cropping area during cropping ensures that the target object is centered within the target image and avoids the problem of the target object being incomplete in the cropped image.

[0026] According to the first aspect, obtaining a first magnification ratio includes: when the initial image is an image captured in an ultra-wide-angle mode, obtaining a second zoom ratio; using the inverse of the second zoom ratio as the first magnification ratio; or, when the initial image is an image captured by the camera at the first zoom ratio, obtaining a ratio of the width of the initial image to the width of the target object annotation box as the first magnification ratio.

[0027] In this way, the first electronic device determines the first magnification ratio according to the second zoom ratio, which can avoid the problem of the enlarged image exceeding the display area; in addition, when the image is captured using the first zoom ratio, it indicates that the captured image is not reduced, and the first magnification ratio can be determined based on the width of the initial image and the target object annotation box, so that the determined first magnification ratio is appropriate.

[0028] According to the first aspect, in the cropping coordinate system, the reference position of the cropping area is determined according to the first magnification ratio, the coordinates of the first key point and the size information of the initial image, including: when it is detected that the difference between the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area is less than or equal to 0, the distance from the reference position of the cropping area to the left border of the initial image is determined to be 0, and the width of the cropping area is the ratio of the width of the initial image to the first magnification ratio; when it is detected that the sum of the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area is greater than or equal to the width of the initial image, the distance from the reference position of the cropping area to the left border of the initial image is determined to be the difference between the width of the initial image and the width of the cropping area; when it is detected that the difference between the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area is greater than 0 and the sum of the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area is less than the width of the initial image, the distance from the reference position of the cropping area to the left border of the initial image is determined to be The distance is the difference between the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area; when it is detected that the difference between the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area is less than or equal to 0, the distance between the reference position of the cropping area and the lower boundary of the initial image is determined to be 0, and the height of the cropping area is the ratio of the height of the initial image to the first magnification ratio; when it is detected that the sum of the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area is greater than or equal to the height of the initial image, the distance between the reference position of the cropping area and the lower boundary of the initial image is determined to be the difference between the height of the initial image and the height of the cropping area; when it is detected that the difference between the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area is greater than 0 and the sum of the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area is less than the height of the initial image, the distance between the reference position of the cropping area and the lower boundary of the initial image is determined to be the difference between the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area.

[0029] Exemplarily, the target object is a human face, and the first key point is the center of the human face.

[0030] Exemplarily, the reference position is taken as the lower left corner of the cropping area.

[0031] In this way, by comparing the difference between the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area with 0, the reference position can be avoided from exceeding the left boundary of the initial image; by detecting whether the sum of the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area is greater than or equal to the width of the initial image, the cropping area can be avoided from exceeding the right boundary of the initial image; by detecting whether the difference between the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area is less than or equal to 0, the reference position can be avoided from exceeding the lower boundary of the initial image, and by detecting whether the sum of the vertical coordinate of the first key point and 1 / 2 of the height of the cropping area is greater than or equal to the height of the initial image, the cropping area can be avoided from exceeding the upper boundary of the initial image; that is, in this example, through the above-mentioned detection, the problem of the cropping area exceeding the initial image can be avoided, ensuring that the target object is centered after cropping.

[0032] According to a first aspect, target image processing is performed on an initial image according to a target shooting mode and target image processing data to obtain a target image, including: when the target shooting mode is a second shooting mode, obtaining a second magnification ratio according to a target area of ​​a target object; determining a reference position of a cropping area in a cropping coordinate system according to the second magnification ratio, coordinates of key points in the target object, and size information of the initial image; converting the reference position in the cropping coordinate system into a reference position in a real image coordinate system; cropping the initial image with the center of the target area as the center of the cropping area and the reference position in the real image coordinate system as the basis for cropping to obtain a second cropped image; amplifying the second cropped image according to the second magnification ratio to obtain a second amplified image; and performing image enhancement processing on the second amplified image to obtain a target image, wherein the clarity of the target image is higher than that of the second amplified image.

[0033] For example, the target object is a human face, and the target area is the area where the eyes, nose or mouth are located.

[0034] In this way, the first electronic device can quickly and accurately determine the reference position of the cropping area based on the second magnification ratio, the coordinates of each key point in the target object, and the size information of the initial image. The initial image is cropped using the center of the target area as the center of the cropping area, and the reference position in the real image coordinate system as the basis for cropping, so that the target area is located at the center of the cropping area. After amplifying the second cropped image, image enhancement processing is performed on the amplified second image to generate the target image, thereby improving the clarity of the target image and avoiding the problem of image blurring caused by amplification.

[0035] According to the first aspect, obtaining the second magnification ratio based on the target area of ​​the target object includes obtaining the quotient of the width of the initial image and the width of the target object annotation box as the second magnification ratio. In this way, the second magnification ratio determined by the first electronic device is more reasonable.

[0036] According to the first aspect, when the target object is a face, a second magnification ratio is obtained based on the target area of ​​the target object, including: when the target area is the left eye or the right eye, obtaining the ratio between the width of the initial image and the distance between the eyes as the second magnification ratio, where the distance between the eyes is the distance between the centers of the left eye and the right eye in the initial image; when the target area is the nose, obtaining the ratio between the width of the initial image and the width of the face as the second magnification ratio, where the width of the face is equal to the width of the target object annotation box; when the target area is the mouth, obtaining the ratio between the width of the initial image and the distance between the eyes as the second magnification ratio or obtaining the ratio between the width of the initial image and the width of the face as the second magnification ratio. In this way, the first electronic device determines different second magnification ratios for different target areas, allowing the first electronic device to more flexibly magnify the target area.

[0037] According to the first aspect, in a cropping coordinate system, a reference position of a cropping area is determined according to a second magnification ratio, coordinates of key points in a target object, and size information of an initial image, including: when it is detected that the difference between the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is less than or equal to 0, determining that the distance between the reference position of the cropping area and the left border of the initial image is 0, the width of the cropping area is the ratio of the width of the initial image to the second magnification ratio, and the target key point is the center point of the target area; when it is detected that the sum of the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is greater than or equal to the width of the initial image, determining that the distance between the reference position of the cropping area and the left border of the initial image is the difference between the width of the initial image and the width of the cropping area; when it is detected that the difference between the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is greater than 0 and the sum of the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is less than the width of the initial image, determining that the distance between the reference position of the cropping area and the left border of the initial image is The distance from the left border of the initial image is the difference between the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area; when it is detected that the difference between the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area is less than or equal to 0, the distance from the reference position of the cropping area to the lower border of the initial image is determined to be 0, and the height of the cropping area is the ratio of the height of the initial image to the second magnification ratio; when it is detected that the sum of the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area is greater than or equal to the height of the initial image, the distance from the reference position of the cropping area to the lower border of the initial image is determined to be the difference between the height of the initial image and the height of the cropping area; when it is detected that the difference between the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area is greater than 0 and the sum of the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area is less than the height of the initial image, the distance from the reference position of the cropping area to the lower border of the initial image is determined to be the difference between the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area.

[0038] For example, the target object is a human face, the target area is the area where the left eye, right eye, nose or mouth are located, and the target key point is the center point of the target area.

[0039] Exemplarily, the reference position is taken as the lower left corner of the cropping area.

[0040] In this way, by comparing the difference between the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area with 0, the reference position can be avoided from exceeding the left boundary of the initial image; by detecting whether the sum of the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is greater than or equal to the width of the initial image, the cropping area can be avoided from exceeding the right boundary of the initial image; by detecting whether the difference between the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area is less than or equal to 0, the reference position can be avoided from exceeding the lower boundary of the initial image, and by detecting whether the sum of the vertical coordinate of the target key point and 1 / 2 of the height of the cropping area is greater than or equal to the height of the initial image, the cropping area can be avoided from exceeding the upper boundary of the initial image; that is, in this example, through the above-mentioned detection, the problem of the cropping area exceeding the initial image can be avoided, ensuring that the target area is centered after cropping.

[0041] According to the first aspect, the reference position in the cropping coordinate system is converted to the reference position in the real image coordinate system, including: obtaining the resolution of the camera, the resolution including the horizontal pixel number of the camera and the vertical pixel number of the camera; obtaining the width and height of the initial image; the distance between the reference position in the real image coordinate system and the left border of the initial image is equal to the quotient of the distance between the reference position in the cropping coordinate system and the left border of the initial image and a first ratio, the first ratio being the ratio of the width of the initial image to the horizontal pixel number of the camera; the distance between the reference position in the real image coordinate system and the lower border of the initial image is equal to the quotient of the distance between the reference position in the cropping coordinate system and the lower border of the initial image and a second ratio, the second ratio being the ratio of the height of the initial image to the vertical pixel number of the camera.

[0042] In this way, through the conversion of the coordinate system, the cropped image is suitable for display on the display screen of the electronic device.

[0043] According to a first aspect, transmitting a target image to a second electronic device, and having the second electronic device display the target image on a second application interface, includes: encoding the target image to obtain encoded data; transmitting the encoded data to the second electronic device, having the second electronic device decode the encoded data to obtain the target image, and displaying the target image on the second application interface of the second electronic device. In this way, the first electronic device encodes the target image before transmitting it, achieving a fast transmission speed and avoiding significant delays when displaying it on the second electronic device.

[0044] According to a first aspect, the first electronic device obtains a target shooting instruction, including: in a case where a second application of a second electronic device calls a camera of the first electronic device, displaying a first interface of a first application in a display screen of the first electronic device, the first interface including a shooting mode selection control; in response to a first operation of the user on the shooting mode selection control in the first interface, displaying a first pop-up box in the first interface, the first pop-up box including a control of a first shooting mode and / or a control of a second shooting mode; in response to a selection operation input by the user in the first pop-up box, generating the target shooting instruction, the selection operation including a first selection operation or a second selection operation.

[0045] In this way, the shooting mode selection control is displayed in the first electronic device, so that the user can perform mode selection on the first electronic device, thereby enhancing the flexibility of user use.

[0046] According to the first aspect, in response to the selection operation input by the user in the first pop-up box, the target shooting instruction is generated, including: in response to the second selection operation input by the user in the first pop-up box, obtaining the second shooting mode as a target shooting mode and displaying a second pop-up box in the first interface, the second pop-up box including a model of a target object, the model of the target object including a plurality of regions to be enlarged; in response to a third selection operation input by the user in the model of the target object, obtaining a region selected by the user from the plurality of regions to be enlarged as a target region; and generating the target shooting instruction according to the target shooting mode and the target region. In this way, the model of the target object is displayed in the first electronic device, which facilitates the user to select the target region for zooming in on the target region. The selection method is intuitive and convenient for user use.

[0047] According to the first aspect, the first electronic device obtains a target shooting instruction, including: in a case where a second application of a second electronic device calls a camera of the first electronic device, the first electronic device receives a target shooting instruction sent by the second electronic device, the target shooting instruction being generated by the second electronic device; wherein a second interface is displayed in a display screen of the second electronic device, the second interface including a shooting mode selection control, the second electronic device displays a third pop-up box in the second interface in response to a second operation of the user on the shooting mode selection control, the third pop-up box including a control of a first shooting mode and / or a control of a second shooting mode; and the second electronic device generates the target shooting instruction in response to a selection operation input by the user in the third pop-up box, the selection operation including a first selection operation or a second selection operation.

[0048] For example, the second interface can be a desktop of the second electronic device.

[0049] In this way, the shooting mode selection control is displayed in the second electronic device, so that the user can perform mode selection on the second electronic device, thereby enhancing the flexibility of user use.

[0050] In a second aspect, the present application provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and, when the computer programs are executed by the one or more processors, cause the electronic device to perform the method for sharing a camera across devices according to the first aspect and any implementation manner of the first aspect.

[0051] The second aspect and any implementation manner of the second aspect correspond to the first aspect and any implementation manner of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation manner of the second aspect can refer to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which will not be described here.

[0052] In a third aspect, the present application provides a computer readable medium for storing a computer program, which, when running on an electronic device, causes the electronic device to perform the method for sharing a camera across devices according to the first aspect and any implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0054] Figure 1a to Figure 1e is a schematic diagram of sharing a camera of a mobile phone by a computer;

[0055] Figure 2 is a schematic diagram of a software framework when a first electronic device and a second electronic device are running;

[0056] Figure 3a is a schematic diagram of the interaction between modules during the process that a computer uses a camera of a mobile phone;

[0057] Figure 3b is a schematic diagram of opening a camera by a DMSDP of a mobile phone;

[0058] Figure 4 is a schematic diagram of the interaction between modules during the process that a computer uses a camera of a mobile phone;

[0059] Figure 5 is a schematic diagram of image processing of an image collected by a mobile phone;

[0060] Figure 6is a schematic diagram of determining the coordinates of the reference position of the cropping, which is exemplarily shown;

[0061] Figure 7a is a schematic diagram of a user selecting a portrait shooting mode, which is exemplarily shown;

[0062] Figure 7b is a schematic diagram of portrait centering, which is exemplarily shown;

[0063] Figure 8 is a schematic diagram of the interaction between modules in the process of the computer using the camera of the mobile phone, which is exemplarily shown;

[0064] Figure 9 is a schematic diagram of the mobile phone performing image processing on the collected image, which is exemplarily shown;

[0065] Figure 10 is a schematic diagram of determining the coordinates of the reference position of the cropping, which is exemplarily shown;

[0066] Figures 11a to 11f is a schematic diagram of a user selecting a macro mode to shoot an image, which is exemplarily shown;

[0067] Figure 12 is a hardware structure diagram of an electronic device, which is exemplarily shown. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0069] The term “and / or” in the present application is only used to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone.

[0070] The terms “first” and “second” and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe the specific order of the target objects.

[0071] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0072] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0073] Before explaining the embodiments of the present application in detail, the application scenarios of the embodiments of the present application will be described with reference to the accompanying drawings. In the embodiments of the present application, both the first electronic device and the second electronic device support interconnection services, and the first electronic device and the second electronic device are connected via the interconnection service. The first electronic device includes a camera and can share the camera with the second electronic device. In this example, the first electronic device is a mobile phone and the second electronic device is a computer.

[0074] The following combination Figure 1a to Figure 1e Specifically explain the process of sharing the phone's camera with the computer.

[0075] When both mobile phone 1 and computer 1 have logged in to the same Honor account, the computer and mobile phone can discover each other and establish a network connection on their own. Figure 1a , computer 1 and mobile phone 1 have established smart interconnection, that is, the computer and mobile phone have established a connection.

[0076] After the computer and mobile phone establish a smart connection, you can enable camera sharing, keyboard and mouse sharing, call sharing and other functions between the computer and mobile phone. Figure 1b and Figure 1c Explain the process of enabling camera sharing on a computer and a mobile phone respectively.

[0077] After the computer and the mobile phone establish a smart interconnection, the user opens the setting interface of the computer's smart interconnection (hereinafter referred to as the first setting interface). In response to the user's opening operation, the computer's display interface displays the first setting interface 101 of the smart interconnection, as shown in FIG. Figure 1b The first setting interface 101 includes a camera sharing control 102, a keyboard and mouse sharing control, and a call sharing control. In response to the user clicking the camera sharing control 102, the first setting interface 101 displays a first control switch 103, as shown. Figure 1b The first control switch is used to control whether to enable the camera sharing function. In response to the user turning on the first control switch 103, the computer turns on the camera sharing function on the computer side.

[0078] like Figure 1c As shown in (1), in response to the user clicking on the setting interface (hereinafter referred to as the second setting interface) in the mobile phone, the display interface of the mobile phone displays the second setting interface 104. The second setting interface 104 includes multiple jump controls, such as the control 1041 for jumping to the smart interconnection. The user clicks on the control 1041, and the mobile phone responds to the user's operation of clicking on the control 1041 and jumps to the setting interface (hereinafter referred to as the third setting interface) 105 of the smart interconnection, such as Figure 1c As shown in (2) in .

[0079] The third setting interface 105 includes a camera sharing control 106, a keyboard and mouse sharing control, and a call sharing control. Optionally, the camera sharing control 106, the keyboard and mouse sharing control, and the call sharing control are initially closed. In response to the user clicking the camera sharing control 106, the interface jumps to the camera sharing control interface 107, such as Figure 1c As shown in (3) in FIG. The camera sharing control interface 107 includes a second control switch 108. Optionally, the initial state of the second control switch is the off state. In response to the user clicking the second control switch 108, the second control switch 108 switches from the off state to the on state, and the mobile phone turns on the camera sharing function, as shown in FIG. Figure 1c As shown in (4).

[0080] When the camera sharing function of the computer and mobile phone is turned on, the computer can take images through the mobile phone's camera, and the mobile phone can also take images through the computer's camera.

[0081] In this example, in response to the user's operation of opening the video conference, the computer starts the video conference, and the computer's display screen displays the video conference interface 109, such as Figure 1d As shown in (1) in . The video conferencing interface 109 includes call duration, microphone control controls, camera control controls 1010, hands-free control controls, and hang-up controls. In response to the user clicking on the camera control control 1010, the computer displays a pop-up window 1011 on the video conferencing interface. Pop-up window 1011 includes: control 1011-1 for "XXX notebook" and control 1011-2 for "XXX mobile phone". If the user clicks on control 1011-1, the computer turns on the camera of the current computer for capturing images. If the user clicks on control 1011-2, the computer turns on the camera in the mobile phone connected to the computer. In this example, in response to the user clicking on control 1011-2, the computer turns on the camera in the mobile phone. As shown in Figure 1dAs shown in (3), the camera of the mobile phone starts to capture images and transmits the captured images back to the computer. When the camera of the mobile phone starts to capture images, the display screen of the mobile phone does not display the captured images, and the display screen of the mobile phone displays a sharing prompt message, which is used to remind the user that the current mobile phone is in camera sharing, such as the sharing prompt message "the mobile phone is in camera sharing". The computer receives the images captured by the mobile phone and displays the images taken by the mobile phone on the interface of the video conference 1012, as shown in FIG. Figure 1d As shown in (2) in .

[0082] When a computer shares a mobile phone's camera, the target object captured by the mobile phone is not located in the center of the captured image, resulting in the target object being off-center in the image displayed on the computer, affecting the user experience (such as the experience of live broadcasting and video conferencing). Figure 1e In the captured image shown in (1), the target person is not in the center. Usually, to solve the problem that the target object captured by the mobile phone is not in the center of the captured image, the user can manually adjust the shooting position of the mobile phone, such as Figure 1e As shown in (2), the user rotates the phone to change the shooting position of the phone. Since the captured image is not displayed on the phone, it is more difficult for the user to manually adjust the shooting position of the phone. As a result, it is difficult to quickly adjust the target object of the phone to the center of the captured image, affecting the user experience.

[0083] In addition, third-party applications with shooting services running on the computer (such as live broadcast applications, instant messaging applications, video conferencing applications, etc.) do not support the operation of zooming in on the captured images, that is, the computer cannot provide users with the function of zooming in on part of the image, affecting user use.

[0084] In view of this, an embodiment of the present application provides a method for sharing a camera across devices, which is applied to a first electronic device, wherein the first electronic device and the second electronic device have established a smart interconnection and the first electronic device has turned on the camera sharing function, and the first electronic device includes a camera. The first electronic device can be a mobile phone, a smart camera or other device, and the second electronic device can be a computer, a tablet computer, etc. When the second electronic device uses the camera of the first electronic device, the second electronic device and the first electronic device provide a shooting mode with the target object centered and / or a macro shooting mode for the target object to meet the need to center the target object in the shooting picture or to partially magnify the target object. The target object can be a person, an animal or an object. The user can select a shooting mode that meets the needs through the first electronic device or the second electronic device.

[0085] The software framework is run in the electronic device to implement the method for sharing a camera across devices provided in the embodiment of the present application. Taking the first electronic device as a mobile phone and the second electronic device as an example, the software frameworks run in the first electronic device and the second electronic device are as follows: Figure 2 shown.

[0086] The software framework related to the method for sharing a camera across devices provided in the embodiment of the present application and running on the computer includes: an application layer, a virtual driver, a device virtualization platform (distributed mobile lesensing development platform, DMSDP), a service interface layer and a service layer.

[0087] The application layer runs various applications, for example, Figure 2 As shown in the figure, applications include camera applications, live broadcast applications, video call applications, etc. These applications all have the function of supporting video services. Camera applications can include native camera applications in computers and third-party camera applications. Video call applications include video call applications, such as The application can also be a program with video conferencing services, such as Honor Conference, etc.

[0088] Virtual drivers include: a virtual bus driver, a virtual camera device, and a virtual microphone device. When a computer and a phone are connected intelligently and camera sharing is enabled on the phone, the phone's camera acts as a virtual camera for the computer. The virtual bus driver on the computer connects to each virtual camera device, driving a specific virtual camera device, such as the phone's camera connected to the computer.

[0089] The device virtualization platform (DMSDP) includes: interface layer, distribution control module, interface security management module, device management module, communication management module, service management module, virtual camera service, virtual audio service, virtual driver operation module, image processing module and codec module.

[0090] The interface layer includes the northbound interface (NBI) and the southbound interface (SBI). The NBI and SBI are interfaces for communicating with upper-layer applications or systems.

[0091] The distribution control module is used to transmit the virtual drive's message to the application layer and receive the message transmitted by the application layer.

[0092] The interface security control module is used to authenticate applications that use camera sharing in the Internet service when the application is an application in the preset list, allowing the authenticated application to use the camera sharing function in the Internet service.

[0093] The device management module manages virtual devices, such as virtual cameras and microphones. The communication management module manages transmission channels. The service management module manages various services on the computer. The virtual camera service provides virtual camera services, while the virtual audio service provides virtual microphone services.

[0094] The virtual driver operation module is used to control the loading and unloading of the virtual driver software.

[0095] The image processing module is used to process the images collected by the computer. The codec module is used to encode or decode the images.

[0096] The business interface layer is used to exchange data with DMSDP. The business layer is used to control the camera's opening and closing instructions.

[0097] The software framework running in the mobile phone and related to the method for sharing a camera across devices provided in the embodiment of the present application includes: a device virtualization platform (distributed mobile lesensing development platform, DMSDP), a service interface layer and a service layer.

[0098] The device virtualization platform (DMSDP) includes: distribution control module, interface security management module, device management module, communication management module, service management module, camera service, image processing module and codec module.

[0099] The distribution control module is used to distribute the driver layer ( Figure 2 The device transmits messages (not shown) to the application layer and receives messages transmitted by the application layer.

[0100] The interface security control module is used to authenticate applications that use camera sharing in the Internet service when the application is an application in the preset list, allowing the authenticated application to use the camera sharing function in the Internet service.

[0101] The device management module manages devices in the phone, such as the camera and microphone. The communication management module manages transmission channels. The service management module manages various services in the computer. The camera service turns the phone's camera on and off.

[0102] The image processing module is used to process the images captured by the mobile phone according to the shooting instructions of the computer to meet the shooting mode selected by the user.

[0103] The codec module is used to encode the image processed by the image processing module and send it to the computer.

[0104] The business interface layer is used to exchange data with DMSDP. The business layer is used to control the camera's opening and closing instructions.

[0105] It is understandable that Figure 2 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0106] It is understandable that in order to implement the shooting method in the embodiment of the present application, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0107] Figure 3a This is a schematic diagram illustrating the interaction between various modules when a computer uses a mobile phone's camera.

[0108] Step S301: The computer video conference generates a first instruction to open the mobile phone camera in response to the operation input by the user.

[0109] For example, computers and mobile phones can Figure 1a The Smart Connectivity service shown in the figure discovers and connects devices. Once a connection is established between a computer and a mobile phone, the two can share applications or devices across devices. For example, a computer can share a phone's call app, a camera, microphone, and other devices. A mobile phone can also share its mouse, keyboard, and other devices. In this example, after a connection is established between a computer and a mobile phone, the computer shares the mobile phone's camera.

[0110] Users can Figure 1c The camera sharing function of the mobile phone is enabled in a manner such that the computer can share the camera of the mobile phone. In response to the user's operation of starting the video conferencing application (i.e., the second application), the computer runs the video conferencing application. In other examples, the computer can start other applications that support shooting functions, such as live broadcast applications, call applications, etc.

[0111] After the computer runs the video conferencing application, the user can Figure 1d As shown in (1), in the pop-up window 1011, click the control 1011-2 to instruct the computer to turn on the camera in the "XXX mobile phone". In response to the user's input operation of selecting the mobile phone camera, the video conferencing application generates a first instruction for instructing to turn on the mobile phone camera.

[0112] It is understandable that in different applications, different startup operations can be set to open the mobile phone camera. For example, the operation input by the user can be a single-click operation, or a sliding, double-clicking operation, etc.

[0113] Step S302: The video conferencing application of the computer sends a first instruction to the virtual driver of the computer.

[0114] For example, when a connection is established between a computer and a mobile phone, when the computer detects that the camera sharing function of the mobile phone has been turned on, the virtual driver operation module of the computer adds a virtual Camera device in the virtual driver module.

[0115] In this example, the video conferencing application of the computer can transmit the first instruction to the virtual driver of the computer, and the virtual bus driver in the virtual driver drives the newly added virtual Camera device to open the camera of the corresponding mobile phone.

[0116] Step S303: The virtual driver of the computer transmits the first instruction to the transmission module of the computer.

[0117] For example, the virtual camera device in the computer's virtual driver transmits a first instruction to the computer's DMSDP. The computer's DMSDP transmits the first instruction to the computer's service interface layer. The computer's service interface layer transmits the first instruction to the computer's service layer, which then transmits the first instruction to the computer's transmission module.

[0118] Step S304: The transmission module of the computer sends the first instruction to the transmission module of the mobile phone.

[0119] Step S305: The transmission module of the mobile phone transmits the first instruction to the DMSDP of the mobile phone.

[0120] Exemplarily, the transmission module of the mobile phone transmits the first instruction to the service layer of the mobile phone, and the service layer of the mobile phone transmits the first instruction to the service interface layer of the mobile phone, and the service interface layer of the mobile phone transmits the first instruction to the DMSDP.

[0121] Step S306: The DMSDP of the mobile phone turns on the camera according to the first instruction.

[0122] For example, the camera service module in the DMSDP of the mobile phone opens the camera (ie, Camera) according to the first instruction. The process of opening the camera by DMSDP can refer to Figure 3b After receiving the first command, the DMSDP sends a command to the camera service to open the camera by calling the camera API2 in the camera framework layer. The camera service sends the open command to the hardware abstraction layer (HAL). The HAL instructs the phone's camera driver to activate the physical camera based on the command.

[0123] Step S307: The DMSDP of the mobile phone obtains the image captured by the camera.

[0124] like Figure 3b As shown, the mobile phone's Camera API 2 obtains each frame of video stream (buffer data) captured by the camera through the callback function (onImageAvailable).

[0125] Step S308: The DMSDP of the mobile phone encodes the collected image.

[0126] As shown in FIG3 , the encoding and decoding module encodes each frame of the acquired video stream.

[0127] Step S309: the DMSDP of the mobile phone transmits the encoded image to the transmission module of the mobile phone.

[0128] Step S310: The transmission module of the mobile phone transmits the encoded image to the transmission module of the computer.

[0129] Step S311: The transmission module of the computer transmits the encoded image to the DMSDP of the computer.

[0130] Exemplarily, the transmission module of the computer transmits the encoded image directly to the DMSDP of the computer.

[0131] Step S312: The DMSDP of the computer decodes the received image.

[0132] The codec module in the DMSDP of the computer decodes the received image to obtain a decoded image.

[0133] Step S313: The DMSDP of the computer transmits the decoded image to the display driver.

[0134] Step S314: the display driver of the computer drives the display screen to display the decoded image.

[0135] Step S315: The computer display screen displays the decoded image.

[0136] In some embodiments, when the user needs the target object to be located in the middle of the captured image, the target object centering mode can be selected on the computer or mobile phone so that the image displayed by the video conferencing application on the computer meets the user's needs. Figure 4 , including the following steps:

[0137] Step S401: The computer's intelligent interconnection service generates a second instruction for centering a target object in response to a user input of a target object centering mode operation.

[0138] For example, the target object is a human face, and the target object centering mode is a portrait centering mode. It is understood that when the target object is an object or an animal, the process of the computer implementing the target object centering mode is similar to this example, and the specific process in this example can be referred to.

[0139] In this example, the user can select the desired shooting mode through the computer-side smart interconnection service to meet their own shooting needs. Optionally, the computer display can display the desktop 701, and the smart service can display the shooting module selection control 702 on the computer desktop, such as Figure 7a As shown in (1) in the figure. The shooting mode selection control 702 is used to provide different shooting effects for the user. For example, in this example, the user clicks the shooting control selection control 702, and a selection box 703 pops up, which includes a portrait centering mode 703-1 and a macro mode 703-2. The operation of the portrait centering mode input by the user can be an operation of the user clicking on the portrait centering mode 703-1. The smart interconnection service generates a second instruction for portrait centering in response to the operation of the portrait centering mode input by the user. The second instruction is used to instruct the mobile phone to obtain a first target image with the portrait centered.

[0140] Optionally, the shooting mode selection control may also be in other styles, such as being displayed on the computer desktop in the form of a floating ball, or being displayed on the computer desktop in the form of a control of a special shape, such as a hexagonal button.

[0141] It should be noted that the shooting mode selection control is displayed outside the interface of the video conferencing application.

[0142] Step S402: The computer's smart interconnection service sends the second instruction to the computer's virtual drive.

[0143] In this example, the computer's smart interconnection service can transmit the second instruction to the computer's virtual driver, and the virtual bus driver in the virtual driver instructs the target virtual Camera device (that is, the camera in the mobile phone currently connected to the computer) to obtain the first target image.

[0144] Step S403: The virtual driver of the computer transmits the second instruction to the transmission module of the computer.

[0145] Exemplarily, the computer's target virtual camera driver transmits the second instruction to the computer's DMSDP. The computer's DMSDP transmits the second instruction to the computer's business interface layer. The computer's business interface layer transmits the second instruction to the computer's business layer, which then transmits the second instruction to the computer's transmission module.

[0146] Step S404: The transmission module of the computer sends the second instruction to the transmission module of the mobile phone.

[0147] Step S405: The transmission module of the mobile phone transmits the second instruction to the DMSDP of the mobile phone.

[0148] This step is similar to step 305. For the specific process, please refer to the relevant description in step 305 and will not be repeated here.

[0149] Step S406: the DMSDP of the mobile phone performs a first image processing on the collected image according to the second instruction to generate a first target image with the target object centered.

[0150] like Figure 5 As shown in FIG, after receiving the second instruction, the DMSDP of the mobile phone obtains each frame of video stream (buffer data) captured by the mobile phone camera by calling the camera interface (Camera api2) in the camera framework layer. The image processing module performs the first image processing on each frame of the video stream to generate the first target image. The process of the image processing module performing the first image processing on each frame of the video stream will be discussed later in conjunction with FIG. Figure 5 Provide specific instructions.

[0151] Step S407: The DMSDP of the mobile phone encodes the first target image to obtain first encoded data.

[0152] Step S408: The DMSDP of the mobile phone transmits the first coded data to the transmission module.

[0153] Step S409: The transmission module of the mobile phone transmits the first encoded data to the transmission module of the mobile phone.

[0154] Step S410: The transmission module of the computer transmits the first encoded data to the DMSDP of the computer.

[0155] Step S411: The DMSDP of the computer decodes the first coded data to obtain a first target image.

[0156] The codec module in the DMSDP of the computer decodes the received image to obtain a decoded image.

[0157] Step S412: The DMSDP of the computer transmits the first target image to the display driver.

[0158] Step S413: The display driver of the computer drives the display screen to display the first target image.

[0159] Step S414: The display screen of the computer displays the first target image.

[0160] Steps 407 to 414 are similar to steps 308 to 315. The relevant process can refer to steps 308 to 315 and will not be repeated here.

[0161] The following combination Figure 5 The process of the image processing module performing the first image processing on each frame of the video stream is described in detail.

[0162] The different shooting modes provided by Smart Connect can be realized by calling the camera interface (Camera api2). In this example, Figure 5 As shown in step 1 of , when Camera api2 receives the second instruction and determines that the current shooting mode is the portrait centered mode, Camera api2 sends each frame of video stream to the face recognition module. Figure 5 As shown in step 2 of FIG, the face recognition module performs face recognition on each frame of the video stream. The face recognition module of the mobile phone is pre-set with a face recognition model, which can be obtained by training based on a deep learning algorithm, such as a convolutional neural network (CNN).

[0163] The training set of the face recognition model may include a first training sample set and a second training sample set, wherein the first training sample set includes different first sample images of face images, and the second training sample set includes second sample images, the second sample images matching the first sample images and having faces annotated in the second sample images. A training device (such as a computer) inputs the first sample image in the first training sample set into a preset training model to obtain training prediction data, compares the training prediction data with the second sample image, and obtains a true value. The training device iteratively adjusts the parameters in the training model based on the true value until the training model converges, thereby obtaining a face recognition model.

[0164] It is understandable that when the target object is an animal, such as a dog or cat, the target object recognition model can be pre-trained by the training equipment. The training process of the target object recognition model can refer to the training process of the face recognition model, which will not be repeated here.

[0165] Continue to refer to Figure 5In step 2 shown, after the face recognition module obtains each frame of the video stream image (hereinafter referred to as an image), a face recognition operation is performed on each frame of the image, that is, the face recognition model identifies the face in each frame of the image. Optionally, the face recognition module can annotate the face in each frame of the image. Optionally, the face annotating method can be to annotate the face with a face rectangular frame on the image, and the face rectangular frame can accurately frame the face area of ​​each face in the image.

[0166] Optionally, in step 2, the face recognition module may only mark the faces in each frame image, so that faces in the image can be quickly identified in subsequent steps. In this example, in step 2, the face recognition module only marks the faces in each frame image as an example for description.

[0167] Optionally, in step 2, the face recognition module can also obtain face width information in each frame image. The face width information can be determined based on the horizontal coordinates of the upper left corner or lower left corner, and the horizontal coordinates of the upper right corner or lower right corner in the face rectangular frame. For example, the coordinates of the upper left corner of the face rectangular frame are (left1, top1), and the coordinates of the lower right corner are (right1, bottom1). For example, if the coordinates of the lower left corner are (left1, bottom2), and the coordinates of the upper right corner are (right1, top2), then the width of the face is rigth1-left1. The current coordinate system can be the real image coordinate system. The face recognition module selects a face with the largest face width from the current image as the target object.

[0168] like Figure 5 In step 3 shown, after the face recognition module identifies a face in each frame, it sets the target interface in Camera API2 to ultra-wide-angle mode. The target interface is the interface for obtaining the camera video stream. The zoom ratio of ultra-wide-angle mode can be set to the maximum value supported by the current phone. For example, if the maximum zoom ratio supported by ultra-wide-angle mode on a current phone is 0.5, the face recognition module sets the target interface mode to ultra-wide-angle mode and the zoom ratio to 0.5.

[0169] like Figure 5 In step 4 shown, Camera API 2 captures images in ultra-wide-angle mode and retransmits the images acquired in ultra-wide-angle mode to the face recognition module. In this example, each frame of image acquired by Camera API 2 in ultra-wide-angle mode is used as a first image.

[0170] The face recognition module can obtain the size information of the first image and the first data of the face in the first image in the coordinate system of the cropping region (i.e., Crop Region). Optionally, the size information of the first image includes the height (such as h) and width (such as w) of the first image in the CropRegion coordinate system. The size information of the first image may also include the center coordinates of the first image, such as (x, y). The first data of the face may include the center coordinates of the face and the coordinates of the target corner in the face rectangular frame. The center coordinates of the face are such as (fx, fy). The target angle is the top angle used to determine the width of the face. For example, the target angle may include the upper left corner and the upper right corner, the upper left corner and the lower right corner, the lower left corner and the upper right corner, and the lower left corner and the lower right corner. In this example, the target angle takes the upper left corner and the lower right corner as an example. The coordinates of the upper left corner of the face rectangular frame are marked as (fLeft, fTop), and the coordinates of the lower right corner are marked as (fRight, fBottom). In this example, the first coordinate (fLeft) of the upper left corner and the first coordinate (fRight) of the lower right corner of the face rectangular box indicate the distance between the vertical border of the face rectangular box and the left / right border of the first image. For example, fLeft indicates the distance between the left border of the face rectangular box and the left border of the first image, and fRight indicates the distance between the right border of the face rectangular box and the left border of the first image; or, fLeft indicates the distance between the left border of the face rectangular box and the right border of the first image, and fRight indicates the distance between the right border of the face rectangular box and the right border of the first image. Similarly, the second coordinate (fTop) of the upper left corner and the second coordinate (fBottom) of the lower right corner of the face rectangular frame indicate the distance between the horizontal border of the face rectangular frame and the upper / lower border of the first image. For example, fTop indicates the distance between the upper border of the face rectangular frame and the upper border of the first image, and fBottom indicates the distance between the lower border of the face rectangular frame and the upper border of the first image; or, fTop indicates the distance between the upper border of the face rectangular frame and the lower border of the first image, and fBottom indicates the distance between the lower border of the face rectangular frame and the lower border of the first image. In this example, fLeft indicates the distance between the left border of the face rectangular frame and the left border of the first image, and fRight indicates the distance between the right border of the face rectangular frame and the left border of the first image, and fTop indicates the distance between the upper border of the face rectangular frame and the upper border of the first image, and fBottom indicates the distance between the lower border of the face rectangular frame and the upper border of the first image.

[0171] Optionally, the face recognition module can determine the width of each face based on the coordinates of the target corners in the face rectangle. For example, the width of the face fSize = fRight-fLeft. When the first image includes multiple face rectangles, the face recognition module traverses the first data of each face. According to the first data of each face, the face recognition module obtains the face with the largest face width as the face to be centered, which is referred to as the centered face or the target face hereinafter.

[0172] As shown in step 5, the image processing submodule obtains the face center coordinates and the face width of the target face from the face recognition module. The image processing submodule can also obtain the size information of the first image from the face recognition module. Figure 5

[0173] As shown in step 6, the image cropping unit in the image processing submodule crops the first image according to the face center coordinates and the face width of the target face. The cropped image is enlarged by the image enlargement unit to make the size of the cropped image equal to the resolution of the current camera. Figure 5

[0174] For example, the image cropping unit takes the face center coordinates of the target face as the cropping center, and determines the reference position of cropping according to the scaling ratio of the ultra-wide-angle mode and the size information of the first image. The image cropping unit crops the first image according to the reference position of cropping and the cropping center to obtain the first cropped image. The reference position of cropping can be any top corner of the four corners of the cropping frame, for example, the reference position of cropping is the lower left corner of the cropping frame, as shown by the dashed box in (a) of Figure 6

[0175] The process of determining the coordinates of the reference position of cropping will be described in detail below. Figure 6 First, the image cropping unit determines the first enlargement ratio of the first cropped image according to the scaling ratio of the ultra-wide-angle mode.

[0176] For example, the image cropping unit can obtain the scaling ratio of the ultra-wide-angle mode, which can be obtained from the face recognition module. The first enlargement ratio can be the inverse of the scaling ratio, for example, the first enlargement ratio is denoted as cropScale, and the scaling ratio is r (r can be 0.5), then the first enlargement ratio cropScale = 1 / r.

[0177] Second, the image cropping unit determines the reference position of the cropping frame.

[0178]

[0179] ​​​​For example, in the Crop Region coordinate system, the width of the cropping frame is represented by x1, and the height of the cropping frame is represented by y1. The cropped image is enlarged according to the first magnification ratio, and the width of the enlarged image is consistent with the width of the first image, and the height of the enlarged image is consistent with the height of the first image. Based on this, the product of the width of the cropping frame and the first magnification ratio is equal to the width of the first image, and the width of the cropping frame is equal to the quotient of the width of the first image and the first magnification ratio, that is, x1 = w / cropScale. Similarly, the height of the cropping frame is equal to the quotient of the height of the first image and the first magnification ratio, that is, y1 = h / cropScale.

[0180] To prevent the target face from being off-center after cropping, the cropping frame needs to be located within the first image. Based on this, the image cropping unit can determine the coordinates of the reference position of the cropping frame. In this example, the reference position is the lower left corner of the cropping frame.

[0181] Figure 6 The coordinate system shown is the Crop Region coordinate system, with the horizontal coordinate of the face center O1 labeled fx and the vertical coordinate of the face center O1 labeled fy. The width of the first image is denoted as w, and the height is denoted as h. The width of the cropping frame is denoted as x1, and the height of the cropping frame is denoted as y1. In the Crop Region coordinate system, the coordinates of the lower left corner of the cropping frame are labeled (cropLeft, cropBottom), where cropLeft indicates the distance between the left border of the cropping frame and the left boundary of the first image, and cropBottom indicates the distance between the bottom border of the cropping frame and the bottom boundary of the first image. The cropping frame is represented by a dotted line, and the first image is represented by a solid black line.

[0182] like Figure 6 As shown in (a), when the difference between the horizontal coordinate of the face center and 1 / 2 of the cropping frame width (hereinafter referred to as the first difference) is equal to 0, the left border of the cropping frame coincides with the left boundary of the first image, that is, cropLeft is equal to 0. When the first difference is less than 0, the left border of the cropping frame will exceed the range of the first image. Therefore, to prevent the left border of the cropping frame from exceeding the range of the first image, when the first difference is less than 0, cropLeft is determined to be equal to 0.

[0183] like Figure 6As shown in (b), when the sum of the horizontal coordinate of the face center coordinate and 1 / 2 of the cropping frame width (hereinafter referred to as the first sum) equals the width of the first image, the right border of the cropping frame coincides with the right boundary of the first image, and cropLeft equals the difference between the width of the first image and the width of the cropping frame (hereinafter referred to as the second difference). When the first sum is greater than the width of the first image, the right border of the cropping frame will extend beyond the range of the first image. To prevent the right border of the cropping frame from extending beyond the range of the first image, when the first sum is greater than the width of the first image, cropLeft equals the second difference.

[0184] like Figure 6 As shown in (c), when neither the first difference is less than 0 nor the first sum is greater than the width of the first image, the cropLeft of the cropping frame is equal to the first difference.

[0185] like Figure 6 As shown in (d), when the difference between the vertical coordinate of the face center (i.e., fy) and half the height of the cropping frame (i.e., y1 / 2) (hereinafter referred to as the third difference) is equal to 0, the bottom border of the cropping frame coincides with the bottom boundary of the first image, that is, cropBottom is equal to 0. When the third difference is less than 0, the bottom border of the cropping frame will exceed the range of the first image. Therefore, to prevent the bottom border of the cropping frame from exceeding the range of the first image, when the third difference is less than 0, cropBottom is determined to be equal to 0.

[0186] like Figure 6 As shown in (e), when the sum of the vertical coordinate of the face center coordinate and 1 / 2 of the cropping frame's height (hereinafter referred to as the second sum) equals the height of the first image, the cropping frame's upper border coincides with the upper boundary of the first image, and cropBottom equals the difference between the height of the first image and the height of the cropping frame (hereinafter referred to as the fourth difference). When the second sum is greater than the width of the first image, the cropping frame's upper border will extend beyond the first image. To prevent the cropping frame's upper border from extending beyond the first image, cropBottom equals the fourth difference when the second sum is greater than the height of the first image.

[0187] like Figure 6 As shown in (f), when neither the third difference is less than 0 nor the second sum is greater than the height of the first image, it is determined that the cropBottom of the cropping frame is equal to the third difference.

[0188] based on Figure 6The image cropping unit can determine the coordinates of the reference position of the cropping frame according to the different positions of the cropping frame shown in . Specifically, the image cropping unit detects whether the difference between the horizontal coordinate of the center of the face and 1 / 2 of the width of the cropping frame (i.e., the first difference) is less than 0. When it is detected that the first difference is less than or equal to 0, cropLeft is determined to be equal to 0. When the sum of the horizontal coordinate of the coordinate of the center of the face and 1 / 2 of the width of the cropping frame (i.e., the first sum) is greater than or equal to the width of the first image, the image cropping unit determines that cropLeft is the difference between the width of the first image and the width of the cropping frame (i.e., the second difference). When it is detected that the first difference is not less than 0 and the first sum is not greater than the width of the first image, cropLeft is determined to be the first difference.

[0189] The image cropping unit detects whether the difference between the vertical coordinate of the center of the face and 1 / 2 of the height of the cropping frame (i.e., the third difference) is less than or equal to 00. When the third difference is detected to be less than or equal to 0, cropBottom is determined to be equal to 0. When the image cropping unit detects that the sum of the vertical coordinate of the center of the face and 1 / 2 of the height of the cropping frame (i.e., the second sum) is greater than or equal to the height of the first image, cropBottom is determined to be the difference between the height of the first image and the height of the cropping frame (i.e., the fourth difference). When the image cropping unit detects that the third difference is not less than 0 and the second sum is not greater than the height of the first image, cropBottom is determined to be the third difference.

[0190] Among them, the code for determining cropLeft can be:

[0191]

[0192] The code to determine cropBottom can be:

[0193]

[0194] Third, the image cropping unit converts the coordinates of the lower left corner of the cropping frame in the Crop Region coordinate system (cropLeft, cropBottom) into the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB). The coordinates of the lower left corner of the cropping region in the real image coordinate system are as follows:

[0195] cropB=cropBottom / (h / mHeight) formula (1);

[0196] cropL=cropLeft / (w / mWidth) formula (2);

[0197] The resolution of the mobile phone's camera is mWidth*mHeight.

[0198] The above three steps are used to determine the coordinates (cropL, cropB) of the lower left corner of the cropping region in the real image coordinate system. After the image cropping unit determines the coordinates (cropL, cropB) of the lower left corner of the cropping region in the real image coordinate system, the first image can be cropped using the coordinates (cropL, cropB) of the lower left corner of the cropping region in the real image coordinate system as the reference position and the coordinates of the center of the face as the cropping center to obtain a first cropped image.

[0199] After acquiring the first cropped image from the image cropping unit, the image magnification unit magnifies the first cropped image according to a first magnification ratio to generate a first target image.

[0200] like Figure 5 In step 7 shown, the image magnification unit in the image processing submodule transmits the first target image to the encoding and decoding module.

[0201] like Figure 5 In step 8 shown, the encoding and decoding module encodes the first target image to generate first encoded data, and transmits the first encoded data to the transmission module of the mobile phone.

[0202] like Figure 5 In step 9 shown, the transmission module of the mobile phone sends the first coded data to the transmission module of the computer.

[0203] After receiving the first coded data, the transmission module on the computer executes the following Figure 4 For the steps 410 to 414 shown, the related descriptions can refer to the steps 410 to 414 and will not be repeated here.

[0204] like Figure 7b As shown, the display interface 701 of the computer terminal displays the shooting mode selection control 702 and the display interface of the video conferencing application, and the display interface of the video conferencing application displays the first target image. The target person in the first target image is centered in the shooting image (i.e., the first target image), and the size of the target person in the first target image is larger than that of the first target image. Figure 7a The target face shown in (1) is large in size on the display interface of the video conferencing application. Figure 7b The clarity of the first target image displayed in the image is not reduced, that is, the clarity of the enlarged image is ensured.

[0205] In another embodiment, when a computer and a mobile phone are connected intelligently and the camera sharing function is enabled on the mobile phone, if the user has already opened the camera on the mobile phone through the computer, the user can also select the target object centering mode on the mobile phone. Figure 7aIn the step (2) shown in FIG. 7, the camera sharing interface 704 is displayed on the display interface of the mobile phone. The camera sharing interface 704 includes a shooting mode selection control 705. The mobile phone pops up a selection box 706 in response to the operation of the user clicking the shooting mode selection control 705. The selection box 706 includes a portrait centering mode 706-1 and a macro mode 706-2. The intelligent interconnection application on the mobile phone generates a second instruction for centering the target object in response to the operation of the user clicking the portrait centering mode 706-1. The intelligent interconnection application of the mobile phone transmits the second instruction to the service layer of the mobile phone, which is transmitted to the service interface layer of the mobile phone. The service interface layer of the mobile phone transmits the second instruction to the DMSDP of the mobile phone. The DMSDP of the mobile phone performs first image processing on the collected image according to the second instruction to generate a first target image (i.e., the image shown in step 406 in FIG. 7). Figure 4 The DMSDP of the mobile phone can subsequently perform steps 407-409, and the computer performs steps 410-414 as described in Figure 4 . Figure 4 The computer performs steps 410-414 as described in .

[0206] In some embodiments, when the user needs to zoom in on a specific region of the target object in the photographed image, the macro mode can be selected on the computer or the mobile phone to make the image displayed by the video conference application on the computer meet the user's needs. The specific process can refer to Figure 8 , including the following steps:

[0207] Step S801: The intelligent interconnection service of the computer generates a third instruction for zooming in on a specified region of the target object in response to the operation of the user inputting the macro mode.

[0208] In the present example, the target object is taken as an example of a face. In the macro mode, the regions to be zoomed in on can be pre-designated to include the eyes, mouth, and nose in the face. It can be understood that when the target object is an object or an animal, the macro mode pre-stores the regions to be zoomed in on, for example, if the target object is a cat, the regions to be zoomed in on can be designated as the eyes, nose, mouth, and ears of the fur. If the target object is a disc-opening bottle, the regions to be zoomed in on can be designated as the bottle opening, bottle body, and bottle bottom of the disc-opening bottle. That is, in the present example, the macro mode can pre-store multiple different regions to be zoomed in on in the target object.

[0209] In the present example, the desktop 1101 of the computer can display a shooting mode selection control 1102 of the intelligent interconnection service, as shown in Figure 11aAs shown in (1), the shooting mode selection control 1102 is used to provide different shooting effects for the user. For example, in this example, the user clicks the shooting control selection control 1102, and a selection box 1103 pops up. The selection box 1103 includes a portrait centering mode 1103-1 and a macro mode 1103-2. The macro mode operation input by the user can be the operation of the user clicking the macro mode 1103-2. Figure 11a As shown in (2), in response to the macro mode operation input by the user, the smart interconnection service pops up a zoom area selection pop-up window 1104. The zoom area selection pop-up window 1104 includes a face model 1105. The user can click on an organ in the face model 1105 to serve as the target organ to be magnified this time. For example, the user clicks on the left eye in the face model 1105. The smart interconnection application generates a third instruction based on the user's selection operation. The third instruction is used to instruct the mobile phone to zoom in on the target area of ​​the target person (the target area is the area where the left eye is located) to obtain an enlarged second target image.

[0210] Optionally, when the target object is other, such as a cat, dog, or bottle, the smart interconnected application displays a general model of the target object, such as a cat model, a dog model, and a bottle model, in the zoom area selection pop-up window. The general model of the target object displays the location that can be designated as the target area so that the user can select the target area to zoom in.

[0211] It should be noted that the shooting mode selection control is displayed outside the interface of the video conferencing application.

[0212] Step S802: The computer's smart interconnection service sends a third instruction to the computer's virtual drive.

[0213] In this example, the intelligent interconnection service of the virtual drive computer can transmit the third instruction to the virtual drive of the computer, and the virtual bus driver in the virtual drive instructs the target virtual Camera device (that is, the camera in the mobile phone currently connected to the computer) to obtain the second target image.

[0214] Step S803: The computer's virtual driver transmits the third instruction to the computer's transmission module.

[0215] Exemplarily, the computer's target virtual camera driver transmits the third instruction to the computer's DMSDP. The computer's DMSDP transmits the third instruction to the computer's service interface layer. The computer's service interface layer transmits the third instruction to the computer's service layer, which then transmits the third instruction to the computer's transmission module.

[0216] Step S804: The transmission module of the computer sends the third instruction to the transmission module of the mobile phone.

[0217] Step S805: The transmission module of the mobile phone transmits the third instruction to the DMSDP of the mobile phone.

[0218] This step is similar to step 305. For the specific process, please refer to the relevant description in step 305 and will not be repeated here.

[0219] Step S806: The DMSDP of the mobile phone performs a second image processing on the collected image according to the third instruction, and generates a second target image of the magnified target area (or designated area).

[0220] like Figure 9 As shown in the figure, after receiving the third instruction, the DMSDP of the mobile phone obtains each frame of video stream (buffer data) captured by the mobile phone camera by calling the camera interface (Camera api2) in the camera framework layer. The image processing module performs the second image processing on each frame of the video stream to generate a second target image. The process of the image processing module performing the second image processing on each frame of the video stream will be discussed later in conjunction with Figure 9 Provide specific instructions.

[0221] Step S807: The DMSDP of the mobile phone encodes the second target image to obtain second encoded data.

[0222] Step S808: The DMSDP of the mobile phone transmits the second encoded data to the transmission module.

[0223] Step S809: The transmission module of the mobile phone transmits the second encoded data to the transmission module of the mobile phone.

[0224] Step S810: The transmission module of the computer transmits the second encoded data to the DMSDP of the computer.

[0225] Step S811: The DMSDP of the computer decodes the second coded data to obtain a second target image.

[0226] The codec module in the DMSDP of the computer decodes the received image to obtain a decoded image.

[0227] Step S812: The DMSDP of the computer transmits the second target image to the display driver.

[0228] Step S813: The display driver of the computer drives the display screen to display the second target image.

[0229] Step S814: The display screen of the computer displays the second target image.

[0230] Steps 807 to 814 are similar to steps 308 to 315. The relevant process can refer to steps 308 to 315 and will not be repeated here.

[0231] The following combination Figure 9 The process of the image processing module performing the second image processing on each frame of the video stream is described in detail.

[0232] The different shooting modes provided by Smart Connect can be realized by calling the camera interface (Camera api2). In this example, Figure 9 As shown in step 1 of , when Camera api2 receives the third instruction and determines that the current shooting mode is macro mode, Camera api2 sends each frame of video stream to the face recognition module. Figure 9 As shown in step 2 of the above, the face recognition module performs face recognition on each video stream. The face recognition module of the mobile phone is pre-set with a face recognition model, which can be obtained by training based on a deep learning algorithm, such as a convolutional neural network (CNN). The training process of the face recognition model can refer to Figure 5 The relevant description in will not be repeated here.

[0233] Continue to refer to Figure 9 In step 2 shown, after the face recognition module obtains each frame of the video stream image (hereinafter referred to as an image), a face recognition operation is performed on each frame of the image, that is, the face recognition model identifies the face in each frame of the image. Optionally, the face recognition module can annotate the face in each frame of the image. Optionally, the face annotating method can be to annotate the face with a face rectangular frame on the image, and the face rectangular frame can accurately frame the face area of ​​each face in the image.

[0234] Optionally, in step 2, the face recognition module may only mark the faces in each frame image, so that faces in the image can be quickly identified in subsequent steps. In this example, in step 2, the face recognition module only marks the faces in each frame image as an example for description.

[0235] Optionally, in step 2, the face recognition module can also obtain face width information in each frame image. The face width information can be determined based on the horizontal coordinates of the upper left corner or lower left corner of the face rectangle, and the horizontal coordinates of the upper right corner or lower right corner. For example, the coordinates of the upper left corner of the face rectangle are (left1, top1), and the coordinates of the lower right corner are (right1, bottom1). For example, if the coordinates of the lower left corner are (left1, bottom2), and the coordinates of the upper right corner are (right1, top2), then the width of the face is rigth1-left1. The current coordinate system can be the real image coordinate system. In this example, the face recognition module selects a face with the largest face width from the current image as the target object.

[0236] In this example, the face recognition module uses each currently acquired frame of image as a frame of the second image.

[0237] like Figure 9 In step 3 shown, the face recognition module can obtain the size information of the second image and the second data of the face in the second image in the crop region (i.e., Crop Region) coordinate system. Optionally, the size information of the second image includes the height (e.g., denoted as h) and width (e.g., denoted as w) of the second image in the Crop Region coordinate system. The size information of the second image can also include the center coordinates of the second image, e.g., denoted as (x, y). The second data of the face can include the center coordinates of the face, the coordinates of the target corners in the face rectangle, and the center coordinates of key organs in the face.

[0238] The face center coordinates are represented as (fx, fy). The target angles are the top corners used to determine the face width. For example, target angles can include the top left and top right corners, the top left and bottom right corners, the bottom left and top right corners, or the bottom left and bottom right corners. In this example, the top left and bottom right corners are used as target angles. The top left corner of the face rectangle is represented as (fLeft, fTop), and the bottom right corner is represented as (fRight, fBottom). In this example, the first coordinate (fLeft) of the upper left corner and the first coordinate (fRight) of the lower right corner of the face rectangular box indicate the distance between the vertical border of the face rectangular box and the left / right border of the first image. For example, fLeft indicates the distance between the left border of the face rectangular box and the left border of the first image, and fRight indicates the distance between the right border of the face rectangular box and the left border of the first image; or, fLeft indicates the distance between the left border of the face rectangular box and the right border of the first image, and fRight indicates the distance between the right border of the face rectangular box and the right border of the first image. Similarly, the second coordinate (fTop) of the upper left corner and the second coordinate (fBottom) of the lower right corner of the face rectangular frame indicate the distance between the horizontal border of the face rectangular frame and the upper / lower border of the first image. For example, fTop indicates the distance between the upper border of the face rectangular frame and the upper border of the first image, and fBottom indicates the distance between the lower border of the face rectangular frame and the upper border of the first image; or, fTop indicates the distance between the upper border of the face rectangular frame and the lower border of the first image, and fBottom indicates the distance between the lower border of the face rectangular frame and the lower border of the first image. In this example, fLeft indicates the distance between the left border of the face rectangular frame and the left border of the first image, and fRight indicates the distance between the right border of the face rectangular frame and the left border of the first image, and fTop indicates the distance between the upper border of the face rectangular frame and the upper border of the first image, and fBottom indicates the distance between the lower border of the face rectangular frame and the upper border of the first image.

[0239] In this example, the key facial features may include the left eye, right eye, mouth, and nose. The center coordinates of the key facial features may include the left eye center coordinates (lEx, lEy), the right eye center coordinates (rEx, rEy), the mouth center coordinates (mx, my), and the nose center coordinates. The nose center coordinates may be equivalent to the face center coordinates, i.e., in this example, the nose center coordinates are equal to the face center coordinates.

[0240] Optionally, the face recognition module can determine the width of each face based on the coordinates of the target corners within the face rectangle, for example, face width fSize = fRight - fLeft. If the first image includes multiple face rectangles, the face recognition module iterates through the second data for each face. Based on the second data for each face, the face recognition module selects the face with the largest face width as the target face.

[0241] like Figure 9 In step 4 shown, the image processing submodule performs second image processing on the second image according to the left eye coordinates and the interocular distance of the target face from the face recognition module.

[0242] like Figure 9 In step 4 shown, the image cropping unit in the image processing submodule crops the second image based on the left eye coordinates and interocular distance of the target face to obtain a second cropped image. The image magnification unit magnifies the second cropped image so that the size of the second cropped image is equal to the resolution of the current camera.

[0243] Exemplarily, the image cropping unit uses the center of the target area of ​​the target face as the cropping center and determines the cropping reference position according to the interocular distance and the size information of the second image. The image cropping unit crops the second image according to the cropping reference position and the cropping center to obtain a second cropped image. The cropping reference position can be any of the four corners of the cropping frame. For example, the cropping reference position is the lower left corner of the cropping frame, such as Figure 10 The lower left corner I2 of the dotted box shown in (a).

[0244] The following combination Figure 10 The image illustrates the process of determining the coordinates of the base position for cropping.

[0245] First, the image cropping unit determines a second magnification ratio of the second cropped image according to the interocular distance.

[0246] For example, the image cropping unit may obtain an interocular distance, where the interocular distance is the distance between the centers of the left eye and the right eye. The second magnification ratio may be the quotient between the width of the second image and the interocular distance. For example, the second magnification ratio is represented by cropScale, and the interocular distance eWidth is adbs(rEx–lEx). Then, the second magnification ratio cropScale=w / eWidth, where w is the width of the current second image and adbs is the sign of the absolute value.

[0247] Second, the image cropping unit determines a reference position of the cropping frame.

[0248] For example, in the Crop Region coordinate system, the width of the cropping frame is represented by x2, and the height of the cropping frame is represented by y2. The cropped image is enlarged according to the second magnification ratio, and the width of the enlarged image is consistent with the width of the second image, and the height of the enlarged image is consistent with the height of the second image. Based on this, the product of the width of the cropping frame and the second magnification ratio is equal to the width of the second image, and the width of the cropping frame is equal to the quotient of the width of the second image and the second magnification ratio, that is, x2 = w / cropScale. Similarly, the height of the cropping frame is equal to the quotient of the height of the second image and the second magnification ratio, that is, y2 = h / cropScale.

[0249] To avoid the cropped target area being incomplete or off-center, the cropping frame needs to be located within the second image. Based on this, the image cropping unit can determine a reference position of the cropping frame. In this example, the reference position is the lower left corner of the cropping frame.

[0250] Figure 10 The coordinate system shown is the Crop Region coordinate system, with the horizontal coordinate of the left eye center O2 labeled LEx and the vertical coordinate of the left eye center O2 labeled LEy. The width of the second image is denoted as w, and the height is denoted as h. The width of the cropping frame is denoted as x2, and the height of the cropping frame is denoted as y2. In the Crop Region coordinate system, the coordinates of the lower left corner of the cropping frame are labeled (cropLeft, cropBottom), where cropLeft indicates the distance between the left border of the cropping frame and the left boundary of the second image, and cropBottom indicates the distance between the bottom border of the cropping frame and the bottom boundary of the second image. The cropping frame is represented by a dotted line, and the second image is represented by a solid black line.

[0251] like Figure 10As shown in (a), when the difference between the horizontal coordinate of the left eye center and 1 / 2 of the cropping frame width (hereinafter referred to as the left eye first detection difference) is equal to 0, the left border of the cropping frame coincides with the left boundary of the second image, that is, cropLeft is equal to 0. When the left eye first detection difference is less than 0, the left border of the cropping frame will exceed the range of the second image. Therefore, to prevent the left border of the cropping frame from exceeding the range of the second image, cropLeft is determined to be 0 when the left eye first detection difference is less than 0.

[0252] like Figure 10 As shown in (b), when the sum of the left eye center's horizontal coordinate and 1 / 2 of the cropping frame's width (hereinafter referred to as the left eye first detection sum) equals the width of the second image, the cropping frame's right border coincides with the second image's right boundary, and cropLeft equals the difference between the second image's width and the cropping frame's width (hereinafter referred to as the left eye second detection difference). When the left eye first detection sum is greater than the second image's width, the cropping frame's right border will extend beyond the second image's bounds. To prevent the cropping frame's right border from extending beyond the second image's bounds, cropLeft equals the left eye second detection difference when the left eye first detection sum is greater than the second image's width.

[0253] like Figure 10 As shown in (c), when neither the first detection difference is less than 0 nor the first detection sum is greater than the width of the second image, the cropLeft of the cropping frame is equal to the left eye first detection difference.

[0254] like Figure 10 As shown in (d), when the difference between the ordinate of the left eye center (i.e., LEy) and half the height of the cropping frame (i.e., y2 / 2) (hereinafter referred to as the left eye third detection difference) is equal to 0, the lower border of the cropping frame coincides with the lower boundary of the second image, that is, cropBottom is equal to 0. When the left eye third detection difference is less than 0, the lower border of the cropping frame will exceed the range of the second image. Therefore, to prevent the lower border of the cropping frame from exceeding the range of the left eye third detection image, cropBottom is determined to be equal to 0 when the left eye third detection difference is less than 0.

[0255] like Figure 10As shown in (e), when the sum of the ordinate of the left eye center coordinate and 1 / 2 of the cropping frame's height (hereinafter referred to as the left eye second detection sum) equals the height of the second image, the cropping frame's upper border coincides with the second image's upper boundary, and cropBottom equals the difference between the second image's height and the cropping frame's height (hereinafter referred to as the left eye fourth detection difference). When the left eye second detection sum is greater than the second image's width, the cropping frame's upper border will extend beyond the second image's bounds. To prevent the cropping frame's upper border from extending beyond the second image's bounds, cropBottom equals the left eye fourth detection difference when the left eye second detection sum is greater than the second image's height.

[0256] like Figure 10 As shown in (f), when neither the left eye third detection difference is less than 0 nor the left eye second detection sum is greater than the height of the second image, the cropBottom of the cropping frame is determined to be equal to the left eye third detection difference.

[0257] based on Figure 10 The image cropping unit can determine the coordinates of the reference position of the cropping frame according to the different positions of the cropping frame shown in . Specifically, the image cropping unit detects whether the difference between the horizontal coordinate of the left eye center and 1 / 2 of the width of the cropping frame (i.e., the left eye first detection difference) is less than 0. When it is detected that the left eye first detection difference is less than or equal to 0, cropLeft is determined to be 0. When the sum of the horizontal coordinate of the left eye center coordinate and 1 / 2 of the width of the cropping frame (i.e., the left eye first detection sum) is greater than or equal to the width of the second image, the image cropping unit determines that cropLeft is the difference between the width of the second image and the width of the cropping frame (i.e., the left eye second detection difference). When it is detected that the left eye first detection difference is not less than 0 and the left eye first detection sum is not greater than the width of the second image, cropLeft is determined to be the left eye first detection difference.

[0258] The image cropping unit detects whether the difference between the vertical coordinate of the left eye center and 1 / 2 of the height of the cropping frame (i.e., the left eye third detection difference) is less than 0. When it is detected that the left eye third detection difference is less than or equal to 0, cropBottom is determined to be equal to 0. When the image cropping unit detects that the sum of the vertical coordinate of the left eye center coordinate and 1 / 2 of the height of the cropping frame (i.e., the left eye second detection sum) is greater than or equal to the height of the second image, cropBottom is determined to be the difference between the height of the second image and the height of the cropping frame (i.e., the left eye fourth detection difference). When the image cropping unit detects that the left eye third detection difference is not less than 0 and the left eye second detection sum is not greater than the height of the second image, cropBottom is determined to be the third detection difference.

[0259] Among them, the code for determining cropLeft can be:

[0260]

[0261] The code to determine cropBottom can be:

[0262]

[0263] Third, the image cropping unit converts the coordinates of the lower left corner of the cropping frame in the Crop Region coordinate system (cropLeft, cropBottom) into the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB). The coordinates of the lower left corner of the cropping region in the real image coordinate system are shown in formulas (1) and (2).

[0264] The above three steps are used to determine the coordinates (cropL, cropB) of the lower left corner of the cropping area in the real image coordinate system. After the image cropping unit determines the coordinates (cropL, cropB) of the lower left corner of the cropping area in the real image coordinate system, the second image can be cropped using the coordinates (cropL, cropB) of the lower left corner of the cropping area in the real image coordinate system as the reference position and the left eye center coordinates as the cropping center to obtain a second cropped image.

[0265] Continue to refer to Figure 9 In step 4 shown, after the image enlarging unit enlarges the second cropped image according to the second enlarging ratio to obtain a second enlarged image, the image enhancing unit performs image enhancement processing on the second enlarged image to generate a second target image.

[0266] The image enhancement unit may be provided with an image enhancement model, which is used to improve the clarity of the input image. The training set of the image enhancement model may include a third training sample set and a fourth training sample set, the third training sample set including a plurality of blurred third sample images, the fourth training sample set including a plurality of fourth sample images, the third sample images matching the fourth sample images and the fourth sample images being clear images. The training device (such as a computer) inputs the third sample image in the third training sample set into a preset second training model to obtain second training prediction data, compares the second training prediction data with the fourth sample image, and obtains a true value. The training device iteratively adjusts the parameters in the second training model based on the true value until the second training model converges to obtain an image enhancement model.

[0267] like Figure 9 In step 7 shown, the image magnification unit in the image processing submodule transmits the second target image to the encoding and decoding module.

[0268] like Figure 9At step 8, the codec module encodes the second target image to generate second encoding data. The codec module transmits the second encoding data to the transmission module of the mobile phone.

[0269] As shown in step 9, the transmission module of the mobile phone transmits the second encoding data to the transmission module of the computer. Figure 9

[0270] After the transmission module of the computer receives the second encoding data, the transmission module of the computer executes steps 810-814 as shown in FIG. 8B, and the related description can refer to steps 810-814, which will not be repeated here. Figure 8

[0271] As shown in FIG. 11A, the display interface 1101 of the computer displays the shooting mode selection control and the display interface 1110 of the video conference application, and the display interface 1110 of the video conference application displays the second target image (the enlarged left eye as shown in the figure). The left eye is displayed in the second target image, and the clarity of the enlarged left eye image is not reduced, that is, the clarity of the enlarged image is ensured. Figure 11c

[0272] In another embodiment, when the computer and the mobile phone are intelligently connected and the camera sharing function of the mobile phone is turned on, if the user has opened the camera of the mobile phone through the computer, the user can also select the macro mode on the mobile phone. Specifically, as shown in (1) of FIG. 11B, the camera sharing interface 1106 is displayed on the display interface of the mobile phone. The camera sharing interface 1106 includes a shooting mode selection control 1107. The mobile phone pops up a selection box in response to the user’s operation of clicking the shooting mode selection control 1107, and the selection box includes a portrait centering mode 1108-1 and a macro mode 1108-2. The intelligent connection application of the mobile phone displays a face model 1109 on the camera sharing interface 1106 in response to the user’s operation of clicking the macro mode 1108-2. The user can click a certain organ in the face model 1109 as the target organ to be enlarged this time. For example, the user clicks the left eye in the face model 1109, and the intelligent connection application generates a third instruction for enlarging the region where the left eye is located according to the user’s selection operation. The third instruction is used to instruct the mobile phone to enlarge the target region of the target person to obtain the second target image of the enlarged left eye. Figure 11b

[0273] The intelligent connection application of the mobile phone transmits the third instruction to the service layer of the mobile phone, and the service layer of the mobile phone transmits the third instruction to the service interface layer of the mobile phone. The service interface layer of the mobile phone transmits the third instruction to the DMSDP of the mobile phone, and the DMSDP of the mobile phone performs second image processing on the collected image according to the third instruction to generate the second target image (that is, the enlarged left eye as shown in the figure). Figure 8 ​​​​The step 806) shown in FIG. 8B. The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment. Figure 8 The step 806) shown in FIG. 8B. The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment. Figure 8 The step 806) shown in FIG. 8B. The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment.

[0274] When the user inputs the macro mode operation from the computer side, the flow of the cross-device camera sharing method provided by the embodiment of the present application refers to the steps in Figure 8 .

[0275] In one embodiment, the smart interconnection application on the computer side displays the zoom-in area selection pop-up window 1104 in response to the user input macro mode operation, as shown in (2) in Figure 11d . The zoom-in area selection pop-up window 1104 includes a face model 1105, and the user can click the right eye in the face model 1105. The smart interconnection application generates a third instruction according to the user's selection operation. The third instruction is used to instruct the mobile phone to zoom in on the target region of the target person (the target region is the region where the left eye is located) to obtain the zoomed-in second target image.

[0276] It should be noted that the user can also select the macro mode on the mobile side and select the target region on the mobile side. The mobile phone pops up a selection box in response to the user's operation of clicking the shooting mode selection control. The selection box includes the portrait centering mode and the macro mode. The smart interconnection application on the mobile side displays the face model 1109 on the camera sharing interface 1106 in response to the user's operation of clicking the macro mode, as shown in (1) in Figure 11d . The user can click the right eye in the face model 1109. The smart interconnection application generates a third instruction for zooming in on the region where the right eye is located according to the user's selection operation. The third instruction is used to instruct the mobile phone to zoom in on the target region of the target person to obtain the zoomed-in second target image (i.e., the second target image). The smart interconnection application of the mobile phone transmits the third instruction to the service layer of the mobile phone, which is transmitted to the service interface layer of the mobile phone. The service interface layer of the mobile phone transmits the third instruction to the DMSDP of the mobile phone. The DMSDP of the mobile phone performs second image processing on the collected image according to the third instruction to generate the second target image (i.e., the step 806) shown in FIG. 8B). The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment. Figure 8 The step 806) shown in FIG. 8B. The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment. Figure 8 The step 806) shown in FIG. 8B. The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment. Figure 8 The step 806) shown in FIG. 8B. The DMSDP on the mobile side can subsequently perform steps 807-809, and the computer side can perform steps 810-814 according to the method described in the foregoing embodiment.

[0277] It should be noted that the specific process of the image cropping unit in the image processing sub-module of the mobile phone to determine the reference position for cropping includes:

[0278] First, the image cropping unit determines a second magnification ratio of the second cropped image according to the interocular distance.

[0279] For example, the image cropping unit can obtain the interocular distance, which is the distance between the center of the left eye and the center of the right eye. The second magnification ratio can be the quotient between the width of the second image and the interocular distance. For example, the second magnification ratio is denoted as cropScale, and the interocular distance eWidth is adbs(rEx-lEx). Then the second magnification ratio cropScale = w / eWidth, where w is the width of the current second image.

[0280] Second, the image cropping unit determines the reference position of the cropping frame.

[0281] The width of the cropping frame is denoted as x3 in the Crop Region coordinate system, and the height of the cropping frame is denoted as y3. The width of the cropping frame is equal to the quotient between the width of the second image and the second magnification ratio, i.e., x3 = w / cropScale. Similarly, the height of the cropping frame is equal to the quotient between the height of the second image and the second magnification ratio, i.e., y3 = h / cropScale, where cropScale is the second magnification ratio.

[0282] To avoid the target region being incomplete or not centered after cropping, the cropping frame needs to be located within the second image. Based on this, the image cropping unit can determine the reference position of the cropping frame. In this example, the reference position is taken as the lower left corner of the cropping frame.

[0283] The image cropping unit detects whether the difference between the horizontal coordinate of the center of the right eye and 1 / 2 of the width of the cropping frame (i.e., the first detection difference of the right eye) is less than 0. When it is detected that the first detection difference of the right eye is less than or equal to 0, it is determined that cropLeft is equal to 0. The image cropping unit detects whether the sum of the horizontal coordinate of the center of the right eye and 1 / 2 of the width of the cropping frame (i.e., the first detection sum of the right eye) is greater than or equal to the width of the second image. When it is detected that the first detection sum of the right eye is greater than or equal to the width of the second image, it is determined that cropLeft is the difference between the width of the second image and the width of the cropping frame (i.e., the second detection difference of the right eye). When it is detected that the first detection difference of the right eye is not less than 0 and the first detection sum of the right eye is not greater than the width of the second image, it is determined that cropLeft is the first detection difference of the right eye.

[0284] The image cropping unit detects whether the difference between the vertical coordinate of the right eye center and 1 / 2 of the height of the cropping frame (i.e., the right eye third detection difference) is less than 0. When it is detected that the right eye third detection difference is less than or equal to 0, cropBottom is determined to be equal to 0. When the image cropping unit detects that the sum of the vertical coordinate of the right eye center coordinate and 1 / 2 of the height of the cropping frame (i.e., the right eye second detection sum) is greater than or equal to the height of the second image, cropBottom is determined to be the difference between the height of the second image and the height of the cropping frame (i.e., the right eye fourth detection difference). When the image cropping unit detects that the right eye third detection difference is not less than 0 and the right eye second detection sum is not greater than the height of the second image, cropBottom is determined to be the right eye third detection difference.

[0285] Among them, the code for determining cropLeft can be:

[0286] cropLeft:

[0287]

[0288] The code to determine cropBottom can be:

[0289]

[0290] The right eye center coordinates are (rEx, rEy), the width of the second image is w, the height of the second image is h, and the second magnification ratio is cropScale, cropScale=w / eWidth, eWidth=adbs(rEx–lEx).

[0291] Third, the image cropping unit converts the coordinates of the lower left corner of the cropping frame in the Crop Region coordinate system (cropLeft, cropBottom) into the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB). The coordinates of the lower left corner of the cropping region in the real image coordinate system are shown in formulas (1) and (2) and are not repeated here.

[0292] The above three steps are used to determine the coordinates (cropL, cropB) of the lower left corner of the cropping area in the real image coordinate system. After the image cropping unit determines the coordinates (cropL, cropB) of the lower left corner of the cropping area in the real image coordinate system, the second image can be cropped using the coordinates (cropL, cropB) of the lower left corner of the cropping area in the real image coordinate system as the reference position and the right eye center coordinates as the cropping center to obtain a second cropped image.

[0293] After the image enlarging unit enlarges the second cropped image according to the second enlarging ratio, a second enlarged image is obtained. The image enhancing unit performs image enhancement processing on the second enlarged image to generate a second target image.

[0294] The effect of enlarging the right eye is as follows Figure 11d As shown in (3), the display interface 1101 of the computer displays a shooting mode selection control and a display interface 1110 of the video conferencing application. The display interface 1111 of the video conferencing application displays a second target image (the enlarged right eye as shown in the figure). The right eye is displayed in the center of the second target image, and the clarity of the enlarged right eye image is not reduced, that is, the clarity of the enlarged image is ensured.

[0295] In one embodiment, the computer-side smart interconnection application responds to the macro mode operation input by the user and displays a zoom area selection pop-up window 1104, such as Figure 11e As shown in (2), the zoom area selection pop-up window 1104 includes a face model 1105. The user can click on the nose in the face model 1105. The smart interconnection application generates a third instruction based on the user's selection operation. The third instruction is used to instruct the mobile phone to zoom in on the target area of ​​the target person to obtain an enlarged second target image. In this example, the target area is the area where the nose is located.

[0296] It should be noted that the user can also select the target area on the mobile phone. In response to the user clicking the macro mode operation, the smart interconnection application on the mobile phone displays the face model 1109 on the camera sharing interface 1106, as shown in FIG. Figure 11e As shown in (1) in the figure. The user can click on the nose in the face model 1109, and the smart interconnection application generates a third instruction based on the user's selection operation. The third instruction is used to instruct the mobile phone to zoom in on the target area of ​​the target person to obtain an enlarged second target image. In this example, the target area is the area where the nose is located. The smart interconnection application of the mobile phone transmits the third instruction to the business layer of the mobile phone, and the business layer of the mobile phone transmits it to the business interface layer of the mobile phone. The business interface layer of the mobile phone transmits the third instruction to the DMSDP of the mobile phone, and the DMSDP of the mobile phone performs a second image processing on the collected image according to the third instruction to generate a second target image (i.e., Figure 8 The DMSDP on the mobile phone can be referred to in the following Figure 8 Execute steps 807 to 809, and the computer will follow Figure 8 Steps 810 to 814 are executed and will not be described again here.

[0297] It should be noted that the specific process of determining the cropping reference position by the image cropping unit in the image processing submodule of the mobile phone includes:

[0298] First, the image cropping unit determines a second magnification ratio of the second cropped image.

[0299] Exemplarily, the image cropping unit can obtain the face width. The face width can be equal to the width of the face rectangle. For example, given the coordinates of the upper left corner (fLeft, fTop) and the lower right corner (fRight, fBottom) of the face rectangle, the face width fWidth = adbs(fRight-fLeft). The second magnification ratio can be the quotient between the width of the second image and the face width. For example, the second magnification ratio is represented by cropScale, and the face width fWidth is adbs(fRight-fLeft). Then the second magnification ratio cropScale = w / fWidth, where w is the width of the current second image.

[0300] Second, the image cropping unit determines a reference position of the cropping frame.

[0301] In the Crop Region coordinate system, the width of the cropping frame is represented by x4, and the height of the cropping frame is represented by y4. The width of the cropping frame is equal to the quotient of the width of the second image and the second magnification ratio, that is, x4 = w / cropScale. Similarly, the height of the cropping frame is equal to the quotient of the height of the second image and the second magnification ratio, that is, y4 = h / cropScale.

[0302] In this example, the lower left corner of the cropping frame is used as the reference position.

[0303] The image cropping unit detects whether the difference between the horizontal coordinate of the face center and 1 / 2 of the width of the cropping frame (i.e., the nose first detection difference) is less than 0. When it is detected that the nose first detection difference is less than or equal to 0, the image cropping unit determines that cropLeft is equal to 0. When it is detected that the sum of the horizontal coordinate of the face center and 1 / 2 of the width of the cropping frame (i.e., the nose first detection sum) is greater than or equal to the width of the second image, the image cropping unit determines that cropLeft is the difference between the width of the second image and the width of the cropping frame (i.e., the nose second detection difference). When it is detected that the nose first detection difference is not less than 0 and the nose first detection sum is not greater than the width of the second image, the image cropping unit determines that cropLeft is the nose first detection difference.

[0304] The image cropping unit detects whether a difference between the vertical coordinate of the face center and 1 / 2 of the height of the cropping frame (i.e., a third detection difference of the nose) is less than 0, and when it is detected that the third detection difference of the nose is less than or equal to 0, it is determined that cropBottom is equal to 0. When the image cropping unit detects that a sum of the vertical coordinate of the face center and 1 / 2 of the height of the cropping frame (i.e., a second detection sum of the nose) is greater than or equal to the height of the second image, it is determined that cropBottom is a difference between the height of the second image and the height of the cropping frame (i.e., a fourth detection difference of the nose). When the image cropping unit detects that the third detection difference of the nose is not less than 0 and the second detection sum of the nose is not greater than the height of the second image, it is determined that cropBottom is the third detection difference of the nose.

[0305] The code for determining cropLeft can be as follows:

[0306] cropLeft:

[0307]

[0308] The code for determining cropBottom can be as follows:

[0309]

[0310] The face center (also the nose center) coordinate is (fx, fy), the width of the second image is w, the height of the second image is h, and the second magnification is cropScale, cropScale = w / fWidth, and fWidth = adbs(fRight-fLeft).

[0311] Third, the image cropping unit converts the coordinates of the lower left corner of the cropping frame in the Crop Region coordinate system (cropLeft, cropBottom) into the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB). The coordinates of the lower left corner of the cropping region in the real image coordinate system are shown in formulas (1) and (2), which will not be described again here.

[0312] The above three steps are used to determine the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB). After the image cropping unit determines the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB), the second image can be cropped based on the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB) as the reference position and the nose center coordinate as the cropping center, to obtain a second cropped image.

[0313] After the image enlarging unit enlarges the second cropped image according to the second enlarging ratio, a second enlarged image is obtained. The image enhancing unit performs image enhancement processing on the second enlarged image to generate a second target image.

[0314] The effect of enlarging the nose is as follows Figure 11e As shown in (3), the display interface 1101 of the computer displays a shooting mode selection control and a display interface 1110 of the video conferencing application, and the display interface 1112 of the video conferencing application displays a second target image (the enlarged nose as shown in the figure).

[0315] In one embodiment, the computer-side smart interconnection application responds to the macro mode operation input by the user and displays a zoom area selection pop-up window 1104, such as Figure 11f As shown in (2), the zoom area selection pop-up window 1104 includes a face model 1105. The user can click on the mouth in the face model 1105. The smart interconnection application generates a third instruction based on the user's selection operation. The third instruction is used to instruct the mobile phone to zoom in on the target area of ​​the target person to obtain an enlarged second target image. In this example, the target area is the area where the mouth is located.

[0316] It should be noted that the user can also select the target area on the mobile phone. In response to the user clicking the macro mode operation, the smart interconnection application on the mobile phone displays the face model 1109 on the camera sharing interface 1106, as shown in FIG. Figure 11f As shown in (1) in the figure. The user can click on the mouth in the face model 1109, and the smart interconnection application generates a third instruction based on the user's selection operation. The third instruction is used to instruct the mobile phone to enlarge the target area of ​​the target person to obtain an enlarged second target image. In this example, the target area is the area where the mouth is located. The smart interconnection application of the mobile phone transmits the third instruction to the business layer of the mobile phone, and the business layer of the mobile phone transmits it to the business interface layer of the mobile phone. The business interface layer of the mobile phone transmits the third instruction to the DMSDP of the mobile phone, and the DMSDP of the mobile phone performs a second image processing on the collected image according to the third instruction to generate a second target image (i.e., Figure 8 The DMSDP on the mobile phone can be referred to in the following Figure 8 Execute steps 807 to 809, and the computer will follow Figure 8 Steps 810 to 814 are executed and will not be described again here.

[0317] It should be noted that the specific process of determining the cropping reference position by the image cropping unit in the image processing submodule of the mobile phone includes:

[0318] First, the image cropping unit determines a second magnification ratio of the second cropped image.

[0319] In one example, the image cropping unit can obtain an interocular distance, which is a distance between a center of the left eye and a center of the right eye. The second scale-up ratio can be a quotient between a width of the second image and the interocular distance, e.g., the second scale-up ratio is denoted as cropScale, the interocular distance eWidth is adbs(rEx–lEx), then the second scale-up ratio cropScale=w / eWidth, where w is a width of the current second image.

[0320] In another example, the image cropping unit can obtain a face width. The face width can be equal to a width of the face rectangle, e.g., given a coordinate of the top-left corner (fLeft, fTop) and a coordinate of the bottom-right corner (fRight, fBottom) of the face rectangle, the face width fWidth=adbs(fRight–fLeft). The second scale-up ratio can also be a quotient between a width of the second image and the face width, e.g., the second scale-up ratio is denoted as cropScale, the face width fWidth is adbs(fRight–fLeft), then the second scale-up ratio cropScale=w / fWidth, where w is a width of the current second image.

[0321] In the present example, the second scale-up ratio is exemplified as a quotient between a width of the second image and the interocular distance.

[0322] Second, the image cropping unit determines a reference position of the cropping frame.

[0323] In the Crop Region coordinate system, a width of the cropping frame is denoted as x5, and a height of the cropping frame is denoted as y5. The width of the cropping frame is equal to a quotient between a width of the second image and the second scale-up ratio, i.e., x5=w / cropScale. Similarly, the height of the cropping frame is equal to a quotient between a height of the second image and the second scale-up ratio, i.e., y5=h / cropScale.

[0324] In the present example, the reference position is exemplified as a bottom-left corner of the cropping frame.

[0325] The image cropping unit detects whether a difference between the horizontal coordinate of the mouth center (i.e., mx) and 1 / 2 of the width of the cropping frame (i.e., a first detected difference of the mouth) is less than 0, and determines that cropLeft is equal to 0 when the first detected difference of the mouth is detected to be less than or equal to 0. The image cropping unit determines that cropLeft is a difference between a width of the second image and a width of the cropping frame (i.e., a second detected difference of the mouth) when a sum of the horizontal coordinate of the mouth center and 1 / 2 of the width of the cropping frame (i.e., a first detected sum of the mouth) is detected to be greater than or equal to the width of the second image. The image cropping unit determines that cropLeft is the first detected difference of the mouth when the first detected difference of the mouth is detected to be not less than 0 and the first detected sum of the mouth is detected to be not greater than the width of the second image.

[0326] The image cropping unit detects whether the difference between the vertical coordinate of the center of the face (i.e., my) and 1 / 2 of the height of the cropping frame (i.e., the third detection difference of the mouth) is less than 0. When the third detection difference of the mouth is detected to be less than or equal to 0, cropBottom is determined to be equal to 0. When the image cropping unit detects that the sum of the vertical coordinate of the center of the mouth and 1 / 2 of the height of the cropping frame (i.e., the second detection sum of the mouth) is greater than or equal to the height of the second image, cropBottom is determined to be the difference between the height of the second image and the height of the cropping frame (i.e., the fourth detection difference of the mouth). When the image cropping unit detects that the third detection difference of the mouth is not less than 0 and the second detection sum of the mouth is not greater than the height of the second image, cropBottom is determined to be the third detection difference of the mouth.

[0327] Among them, the code for determining cropLeft can be:

[0328] cropLeft:

[0329]

[0330] The code to determine cropBottom can be:

[0331]

[0332] The coordinates of the center of the mouth are (mx, my), the width of the second image is w, the height of the second image is h, and the second magnification ratio is cropScale, cropScale=w / eWidth, eWidth=adbs(rEx–lEx).

[0333] Third, the image cropping unit converts the coordinates of the lower left corner of the cropping frame in the Crop Region coordinate system (cropLeft, cropBottom) into the coordinates of the lower left corner of the cropping region in the real image coordinate system (cropL, cropB). The coordinates of the lower left corner of the cropping region in the real image coordinate system are shown in formulas (1) and (2) and are not repeated here.

[0334] The above three steps are used to determine the coordinates (cropL, cropB) of the lower left corner of the cropping region in the real image coordinate system. After the image cropping unit determines the coordinates (cropL, cropB) of the lower left corner of the cropping region in the real image coordinate system, the second image can be cropped using the coordinates (cropL, cropB) of the lower left corner of the cropping region in the real image coordinate system as the reference position and the coordinates of the center of the mouth as the cropping center to obtain a second cropped image.

[0335] After the image enlarging unit enlarges the second cropped image according to the second enlarging ratio, a second enlarged image is obtained. The image enhancing unit performs image enhancement processing on the second enlarged image to generate a second target image.

[0336] The effect of enlarging the mouth is as follows Figure 11f As shown in (3), the display interface 1101 of the computer displays a shooting mode selection control and a display interface 1110 of the video conferencing application, and the display interface 1113 of the video conferencing application displays a second target image (the enlarged mouth as shown in the figure).

[0337] It should be noted that the electronic devices to which the embodiments of the present application are applicable may be, for example, mobile phones and computers.

[0338] It should be understood that Figure 12 The illustrated electronic device 100 is merely one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the drawings, may combine two or more components, or may have a different configuration of components. Figure 12 The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits. In this example, the electronic device 100 is taken as an example of a mobile phone.

[0339] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a Subscriber Identification Module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0340] It can be understood that, in order to achieve the above functions, the electronic device comprises hardware and / or software modules corresponding to the functions. The algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0341] The embodiments of the present application also provide a chip system, which comprises at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a line. For example, the interface circuit can be used to receive signals from other devices (such as the memory of the electronic device). For another example, the interface circuit can be used to send signals to other devices (such as the processor). Illustratively, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform each step in the above embodiments. Of course, the chip system can also comprise other discrete devices, which are not limited in the embodiments of the present application.

[0342] The embodiments of the present application also provide a computer storage medium, which stores computer instructions. When the computer instructions are run on the electronic device, the electronic device performs the above related method steps to implement the method for sharing the camera across devices in the above embodiments. The storage medium includes a U disk, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0343] The embodiments of the present application also provide a computer program product, which, when run on a computer, causes the computer to perform the above related steps to implement the method for sharing the camera across devices in the above embodiments.

[0344] In the embodiments of the present application, the electronic device, the computer storage medium, the computer program product or the chip are used to perform the above corresponding method for sharing the camera across devices, and thus the beneficial effects achieved by the above corresponding method can be referred to, and will not be described herein.

[0345] Any content of each embodiment of the present application, and any content of the same embodiment, can be freely combined. Any combination of the above is within the scope of the present application.

[0346] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for sharing a camera across devices, characterized in that: The method is applied to a first electronic device, wherein the first electronic device and the second electronic device each have a first application installed thereon, and the first electronic device establishes a communication connection with the second electronic device through the first application. The method includes: In a case where a second application of the second electronic device calls a camera of the first electronic device, the first electronic device obtains a target shooting instruction, where the target shooting instruction is generated based on a target shooting mode selected by a user through a first application of the first electronic device or a first application of the second electronic device, where the first application is different from the second application; wherein the shooting mode to be selected includes a first shooting mode and / or a second shooting mode, the first shooting mode is used to indicate that a target object is centered in a target image, and the second shooting mode is used to indicate that a target area of ​​the target object is magnified and displayed in the target image; capturing an image in accordance with the target capturing mode indicated by the target capturing instruction to obtain a target image; The target image is transmitted to the second electronic device, and the second electronic device displays the target image on a second application interface.

2. The method according to claim 1, characterized in that Shooting an image in accordance with the target shooting mode indicated by the target shooting instruction to obtain the target image includes: In response to the target shooting instruction, acquiring an image shot by the first electronic device as an initial image; Identify the target object in the initial image; When the target object is identified, acquiring image processing data of the initial image according to the target shooting mode and the recognition result, the image processing data including size information of the initial image and coordinates of key points of the target object; According to the target shooting mode and the image processing data, target image processing is performed on the initial image to obtain a target image, wherein the target image processing includes cropping the initial image and enlarging the cropped image.

3. The method according to claim 2, characterized in that In response to the target shooting instruction, acquiring an image shot by the first electronic device as an initial image includes: In response to the target shooting instruction, acquiring an image captured by a camera at a first zoom rate as an initial image; When the target shooting instruction indicates that the target shooting mode is the first shooting mode, after identifying the target object in the initial image, the method further includes: Adjusting the zoom ratio of the camera to a second zoom ratio corresponding to the ultra-wide-angle mode; Acquire an image captured in the ultra-wide-angle mode; The initial image is updated with the image captured in the ultra-wide-angle mode.

4. The method according to claim 3, characterized in that When the target object is identified, obtaining image processing data of the initial image according to the target shooting mode and the recognition result includes: When the target shooting mode is the first shooting mode, obtaining size information of the initial image, obtaining coordinates of a first key point of the target object, and obtaining a width of the target object annotation box in a cropping coordinate system, where the first key point is a center point of the target object; When the target shooting mode is the second shooting mode, the size information of the initial image, the coordinates of each key point in the target object and the width of the target object annotation box are obtained in the cropping coordinate system, where the key points of the target object include the center points of each preset area to be magnified in the target object.

5. The method according to claim 4, characterized in that Performing target image processing on the initial image according to the target shooting mode and the target image processing data to obtain the target image includes: When the target shooting mode is the first shooting mode, obtaining a first magnification ratio; In a cropping coordinate system, determining a reference position of a cropping area according to the first magnification ratio, the coordinates of the first key point, and the size information of the initial image; Converting the reference position in the clipping coordinate system into a reference position in the real image coordinate system; Using the first key point as the center of a cropping area and the reference position in the real image coordinate system as a reference for cropping, the initial image is cropped to obtain a first cropped image; The first cropped image is enlarged according to the first enlargement ratio to obtain the target image.

6. The method according to claim 5, characterized in that Obtaining a first magnification ratio includes: When the initial image is an image captured in the ultra-wide-angle mode, obtaining the second zoom ratio; Using the inverse of the second zoom ratio as the first magnification ratio; or, When the initial image is an image captured by the camera at the first zoom ratio, a ratio of a width of the initial image to a width of the target object annotation box is obtained as the first magnification ratio.

7. The method according to claim 6, characterized in that The determining, in the cropping coordinate system, a reference position of the cropping area according to the first magnification ratio, the coordinates of the first key point, and the size information of the initial image includes: When it is detected that the difference between the horizontal coordinate of the first key point and 1 / 2 of the width of the cropping area is less than or equal to 0, determining that the distance between the reference position of the cropping area and the left border of the initial image is 0, and the width of the cropping area is the ratio of the width of the initial image to the first magnification ratio; When it is detected that the sum of the abscissa of the first key point and 1 / 2 of the width of the cropping area is greater than or equal to the width of the initial image, determining that the distance between the reference position of the cropping area and the left border of the initial image is the difference between the width of the initial image and the width of the cropping area; When it is detected that the difference between the abscissa of the first key point and 1 / 2 of the width of the cropping area is greater than 0 and the sum of the abscissa of the first key point and 1 / 2 of the width of the cropping area is less than the width of the initial image, determining that the distance between the reference position of the cropping area and the left border of the initial image is the difference between the abscissa of the first key point and 1 / 2 of the width of the cropping area; When it is detected that the difference between the longitudinal coordinate of the first key point and 1 / 2 of the height of the cropping area is less than or equal to 0, determining that the distance between the reference position of the cropping area and the lower boundary of the initial image is 0, and the height of the cropping area is the ratio of the height of the initial image to the first magnification ratio; When it is detected that the sum of the longitudinal coordinate of the first key point and 1 / 2 of the height of the cropping area is greater than or equal to the height of the initial image, determining that the distance between the reference position of the cropping area and the lower boundary of the initial image is the difference between the height of the initial image and the height of the cropping area; When it is detected that the difference between the longitudinal coordinate of the first key point and 1 / 2 of the height of the cropping area is greater than 0 and the sum of the longitudinal coordinate of the first key point and 1 / 2 of the height of the cropping area is less than the height of the initial image, the distance between the reference position of the cropping area and the lower boundary of the initial image is determined to be the difference between the longitudinal coordinate of the first key point and 1 / 2 of the height of the cropping area.

8. The method according to claim 4, characterized in that Performing target image processing on the initial image according to the target shooting mode and the target image processing data to obtain the target image includes: When the target shooting mode is the second shooting mode, obtaining a second magnification ratio according to the target area of ​​the target object; determining, in a cropping coordinate system, a reference position of the cropping area according to the second magnification ratio, coordinates of key points in the target object, and size information of the initial image; Converting the reference position in the clipping coordinate system to the reference position in the real image coordinate system; Using the center of the target area as the center of the cropping area and the reference position in the real image coordinate system as the reference for cropping, the initial image is cropped to obtain a second cropped image; enlarging the second cropped image according to the second enlargement ratio to obtain a second enlarged image; Performing image enhancement processing on the second enlarged image to obtain the target image, wherein the clarity of the target image is higher than the clarity of the second enlarged image.

9. The method according to claim 8, characterized in that Acquiring a second magnification ratio according to a target area of ​​the target object includes: The quotient of the width of the initial image and the width of the target object annotation box is obtained as the second magnification ratio.

10. The method according to claim 8, characterized in that In the case where the target object is a human face; Acquiring a second magnification ratio according to a target area of ​​the target object includes: When the target area is the left eye or the right eye, obtaining a ratio between a width of the initial image and an interocular distance as a second magnification ratio, where the interocular distance is the distance between the center of the left eye and the center of the right eye in the initial image; When the target area is a nose, obtaining a ratio between a width of the initial image and a width of the face as the second magnification ratio, the width of the face being equal to a width of the target object annotation box; When the target area is a mouth, a ratio between the width of the initial image and the interocular distance is obtained as the second magnification ratio, or a ratio between the width of the initial image and the width of the face is obtained as the second magnification ratio.

11. The method according to claim 9 or 10, characterized in that In the cropping coordinate system, determining a reference position of the cropping area according to the second magnification ratio, the coordinates of each key point in the target object, and the size information of the initial image includes: When it is detected that the difference between the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is less than or equal to 0, determining that the distance between the reference position of the cropping area and the left border of the initial image is 0, the width of the cropping area is the ratio of the width of the initial image to the second magnification ratio, and the target key point is the center point of the target area; When it is detected that the sum of the horizontal coordinate of the target key point and 1 / 2 of the width of the cropping area is greater than or equal to the width of the initial image, determining that the distance between the reference position of the cropping area and the left border of the initial image is the difference between the width of the initial image and the width of the cropping area; When it is detected that the difference between the abscissa of the target key point and 1 / 2 of the width of the cropping area is greater than 0 and the sum of the abscissa of the target key point and 1 / 2 of the width of the cropping area is less than the width of the initial image, determining that the distance between the reference position of the cropping area and the left border of the initial image is the difference between the abscissa of the target key point and 1 / 2 of the width of the cropping area; When it is detected that the difference between the longitudinal coordinate of the target key point and 1 / 2 of the height of the cropping area is less than or equal to 0, determining that the distance between the reference position of the cropping area and the lower boundary of the initial image is 0, and the height of the cropping area is the ratio of the height of the initial image to the second magnification ratio; When it is detected that the sum of the ordinate of the target key point and 1 / 2 of the height of the cropping area is greater than or equal to the height of the initial image, determining that the distance between the reference position of the cropping area and the lower boundary of the initial image is the difference between the height of the initial image and the height of the cropping area; When it is detected that the difference between the longitudinal coordinate of the target key point and 1 / 2 of the height of the cropping area is greater than 0 and the sum of the longitudinal coordinate of the target key point and 1 / 2 of the height of the cropping area is less than the height of the initial image, the distance between the reference position of the cropping area and the lower boundary of the initial image is determined to be the difference between the longitudinal coordinate of the target key point and 1 / 2 of the height of the cropping area.

12. The method according to claim 5 or 8, characterized in that Converting the reference position in the clipping coordinate system to a reference position in the real image coordinate system includes: Obtaining the resolution of the camera, where the resolution includes the number of horizontal pixels of the camera and the number of vertical pixels of the camera; Get the width and height of the initial image; The distance between the reference position in the real image coordinate system and the left border of the initial image is equal to the quotient of the distance between the reference position in the cropping coordinate system and the left border of the initial image and a first ratio, where the first ratio is the ratio of the width of the initial image to the number of horizontal pixels of the camera; The distance between the reference position in the real image coordinate system and the lower boundary of the initial image is equal to the quotient of the distance between the reference position in the cropping coordinate system and the lower boundary of the initial image and a second ratio, where the second ratio is the ratio of the height of the initial image to the number of vertical pixels of the camera.

13. The method according to claim 1, wherein The first electronic device obtains a target shooting instruction, including: In a case where a second application of the second electronic device calls a camera of the first electronic device, displaying a first interface of the first application on a display screen of the first electronic device, where the first interface includes a shooting mode selection control; In response to a first operation of the user on the shooting mode selection control on the first interface, displaying a first pop-up window on the first interface, where the first pop-up window includes controls for the first shooting mode and / or controls for the second shooting mode; In response to a selection operation input by the user in the first pop-up box, a target shooting instruction is generated, where the selection operation includes a first selection operation or a second selection operation.

14. The method according to claim 13, wherein: In response to a selection operation input by the user in the first pop-up window, generating a target shooting instruction includes: In response to a second selection operation input by the user in the first pop-up window, acquiring the second shooting mode as the target shooting mode and displaying a second pop-up window on the first interface, wherein the second pop-up window includes a model of the target object, and the model of the target object includes multiple areas to be magnified; In response to a user inputting a third selection operation in the model of the target object, obtaining an area selected by the user from a plurality of areas to be magnified as the target area; The target shooting instruction is generated according to the target shooting mode and the target area.

15. The method according to claim 1, wherein The first electronic device obtains a target shooting instruction, including: In a case where the second application of the second electronic device calls the camera of the first electronic device, the first electronic device receives the target shooting instruction sent by the second electronic device, where the target shooting instruction is generated by the second electronic device; In which, a second interface is displayed on the display screen of the second electronic device, and the second interface includes a shooting mode selection control. The second electronic device responds to the user's second operation on the shooting mode selection control, and displays a third pop-up box on the second interface. The third pop-up box includes controls for the first shooting mode and / or controls for the second shooting mode; the second electronic device generates a target shooting instruction in response to the user's selection operation input in the third pop-up box, and the selection operation includes the first selection operation or the second selection operation.

16. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the method for sharing a camera across devices as described in any one of claims 1 to 15.

17. A computer-readable storage medium comprising a computer program, characterized in that When the computer program is run on an electronic device, the electronic device is enabled to execute the method for sharing a camera across devices as described in any one of claims 1 to 15.

Images