Camera cooperative picture synthesis method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a camera cooperative picture synthesis method and device, electronic equipment and storage medium. The method comprises the following steps: synchronizing a control instruction of a local camera module to a called device, wherein the control instruction is used to control the local camera module of the electronic equipment to acquire a local image and control a remote camera module of the called device to acquire a remote image; acquiring the remote image returned by the called device; synthesizing the local image and the remote image to obtain a synthesis image; and displaying the synthesis image on the electronic equipment. In this embodiment, the local image and the remote image are synthesized and displayed, multi-path pictures and multi-view display are realized, the advantages of camera cooperative display can be played, and the use experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, and in particular to a method, apparatus, electronic device and storage medium for camera-coordinated image synthesis. Background Technology

[0002] In multi-device networking application scenarios, there is a need for one device to use the camera hardware of other devices (hereinafter referred to as camera collaboration function). For example, an electronic device (hereinafter referred to as electronic device or calling device) can directly call the camera on another electronic device (hereinafter referred to as the called device) to transmit images.

[0003] In the existing solution, after the camera collaboration function is enabled, the display screen of the camera application on the electronic device will automatically switch to the screen captured by the called device, resulting in the loss of the screen captured by the local camera. Summary of the Invention

[0004] This disclosure provides a camera-coordinated image composition method, apparatus, electronic device, and storage medium to solve the aforementioned technical problems.

[0005] According to a first aspect of this disclosure, a camera-coordinated image composition method is provided, the method comprising:

[0006] The control commands of the local camera module are synchronized to the called device. The control commands are used to control the local camera module of the electronic device to acquire local images and to control the remote camera module of the called device to acquire remote images.

[0007] Obtain the remote image returned by the called device;

[0008] The local image and the remote image are combined to obtain a composite image;

[0009] The composite image is displayed on an electronic device.

[0010] Optionally, control commands from the local camera module are synchronized to the called device, including:

[0011] In response to detecting a camera activation request sent by a third-party camera application, the camera activation request is synchronized to the remote camera module of the invoked device.

[0012] In response to detecting a data stream configuration request sent by the third-party camera application, the data stream configuration request is synchronized to the remote camera module;

[0013] In response to detecting an image processing capture request sent by a third-party camera application, the image processing capture request is synchronized to the remote camera module.

[0014] Optionally, synchronizing the camera activation request to the remote camera module of the invoked device includes:

[0015] When the camera injection manager of the electronic device detects that the third-party camera application calls the camera start interface, it determines that it has obtained the camera start request.

[0016] The camera activation request is passed through to the virtual camera service;

[0017] The virtual camera service sends the camera activation request to the invoked device; the camera activation request is used to instruct the virtual camera application of the invoked device to attempt to activate the remote camera module.

[0018] Optionally, synchronizing the data stream configuration request to the remote camera module includes:

[0019] When the camera injection manager of the electronic device detects that the third-party camera application requests to call the camera data stream interface, it determines that it has obtained the data stream configuration request;

[0020] The data stream configuration request is passed through to the virtual camera service;

[0021] The virtual camera service sends the data stream configuration request to the invoked device; the data stream configuration request is used to instruct the virtual camera application of the invoked device to attempt to request the corresponding data stream from the already activated camera.

[0022] Optionally, synchronizing the image processing capture request to the remote camera module includes:

[0023] When the camera injection manager of the electronic device detects that the third-party camera application requests to call the interface to initiate a preview or image processing capture request, it determines that the image processing capture request has been obtained.

[0024] The image processing capture request is passed through to the virtual camera service;

[0025] The virtual camera service sends the image processing and capture request to the invoked device; the image processing and capture request is used to instruct the virtual camera application of the invoked device to send a request to start previewing or take a picture.

[0026] Optionally, the local image and the remote image are combined to obtain a composite image, including:

[0027] Obtain the image data type requested by the image processing capture request within the control command, wherein the image data type includes preview type or photo type;

[0028] The image configuration format of the synthesized image is obtained, including an image side-by-side format or a picture-in-picture format;

[0029] The local image and the remote image are processed according to the image configuration format to obtain a composite image that matches the image data type.

[0030] Optionally, obtaining the remote image returned by the called device includes:

[0031] The virtual camera service within the camera hardware abstraction layer of the electronic device acquires the remote image fed back by the invoked device;

[0032] The virtual camera service sends the remote image to the virtual camera;

[0033] The virtual camera sends the remote image to the camera injection session to obtain the remote image captured by the remote camera module.

[0034] Optionally, displaying the synthesized image on an electronic device includes:

[0035] The stream injection session of the camera hardware abstraction layer within the electronic device returns the synthesized image to the camera injection session.

[0036] The camera injection session forwards the composite image to the camera injection manager;

[0037] The camera injection manager replaces the local image with the synthesized image;

[0038] The camera injection manager passes the synthesized image through to a third-party camera application.

[0039] According to a second aspect of this disclosure, a camera-coordinated image synthesis apparatus is provided, the apparatus comprising:

[0040] The camera hardware abstraction layer is used to synchronize control commands from the local camera module to the called device. The control commands are used to control the local camera module of the electronic device to acquire local images and to control the remote camera module of the called device to acquire remote images.

[0041] The camera hardware abstraction layer is used to synthesize the local image and the remote image to obtain a synthesized image;

[0042] The camera hardware abstraction layer is used to display the synthesized image on an electronic device.

[0043] Optionally, the camera hardware abstraction layer synchronizes control commands from the local camera module to the invoked device, including:

[0044] The camera hardware abstraction layer is used to respond to a camera activation request sent by a third-party camera application and synchronize the camera activation request to the remote camera module of the invoked device.

[0045] The camera hardware abstraction layer is used to synchronize the data stream configuration request to the remote camera module in response to detecting the data stream configuration request sent by the third-party camera application.

[0046] The camera hardware abstraction layer is used to synchronize the image processing and capture request to the remote camera module in response to the detection of an image processing and capture request sent by a third-party camera application.

[0047] Optionally, the camera hardware abstraction layer synchronizes the camera activation request to the remote camera module of the invoked device, including:

[0048] When the camera injection manager of the electronic device detects that the third-party camera application calls the camera start interface, it determines that it has obtained the camera start request.

[0049] The camera activation request is passed through to the virtual camera service;

[0050] The virtual camera service sends the camera activation request to the invoked device; the camera activation request is used to instruct the virtual camera application of the invoked device to attempt to activate the remote camera module.

[0051] Optionally, the camera hardware abstraction layer synchronizes the data stream configuration request to the remote camera module, including:

[0052] When the camera injection manager of the electronic device detects that the third-party camera application requests to call the camera data stream interface, it determines that it has obtained the data stream configuration request;

[0053] The data stream configuration request is passed through to the virtual camera service;

[0054] The virtual camera service sends the data stream configuration request to the invoked device; the data stream configuration request is used to instruct the virtual camera application of the invoked device to attempt to request the corresponding data stream from the already activated camera.

[0055] Optionally, the camera hardware abstraction layer synchronizes the image processing capture request to the remote camera module, including:

[0056] When the camera injection manager of the electronic device detects that the third-party camera application requests to call the interface to initiate a preview or image processing capture request, it determines that the image processing capture request has been obtained.

[0057] The image processing capture request is passed through to the virtual camera service;

[0058] The virtual camera service sends the image processing and capture request to the invoked device; the image processing and capture request is used to instruct the virtual camera application of the invoked device to send a request to start previewing or take a picture.

[0059] Optionally, the camera hardware abstraction layer synthesizes the local image and the remote image to obtain a synthesized image, including:

[0060] Obtain the image data type requested by the image processing capture request within the control command, wherein the image data type includes preview type or photo type;

[0061] The image configuration format of the synthesized image is obtained, including an image side-by-side format or a picture-in-picture format;

[0062] The local image and the remote image are processed according to the image configuration format to obtain a composite image that matches the image data type.

[0063] Optionally, the camera hardware abstraction layer acquires the remote image returned by the called device, including:

[0064] The virtual camera service within the camera hardware abstraction layer of the electronic device acquires the remote image fed back by the invoked device;

[0065] The virtual camera service sends the remote image to the virtual camera;

[0066] The virtual camera sends the remote image to the camera injection session to obtain the remote image captured by the remote camera module.

[0067] Optionally, the camera hardware abstraction layer displays the synthesized image on an electronic device, including:

[0068] The stream injection session of the camera hardware abstraction layer within the electronic device returns the synthesized image to the camera injection session.

[0069] The camera injection session forwards the composite image to the camera injection manager;

[0070] The camera injection manager replaces the local image with the synthesized image;

[0071] The camera injection manager passes the synthesized image through to a third-party camera application.

[0072] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor and a memory;

[0073] The memory is used to store computer programs that can be executed by the processor;

[0074] The processor is configured to execute a computer program in the memory to implement the method as described in any of the first aspects.

[0075] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method as described in any of the first aspects.

[0076] According to a fifth aspect of this disclosure, a non-transitory computer-readable storage medium is provided, which, when an executable computer program in the storage medium is executed by a processor, enables the implementation of the method as described in any of the first aspects.

[0077] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0078] The solution provided in this embodiment first synchronizes the control commands of the local camera module to the called device. These control commands control the local camera module of the electronic device to acquire local images and control the remote camera module of the called device to acquire remote images. Then, the remote image returned by the called device is acquired. Next, the local image and the remote image are combined to obtain a composite image. Finally, the composite image is displayed on the electronic device. Thus, this embodiment combines and displays local and remote images, achieving multi-view and multi-angle display, leveraging the advantages of camera collaborative display, and improving the user experience.

[0079] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0080] Figure 1 This is a flowchart of a camera-coordinated image synthesis method according to an embodiment of the present disclosure.

[0081] Figure 2 This is a flowchart of a synchronization control instruction according to an embodiment of the present disclosure.

[0082] Figure 3 This is a schematic diagram of an image side-by-side format according to an embodiment of the present disclosure.

[0083] Figure 4 This is a schematic diagram of a picture-in-picture format according to an embodiment of the present disclosure.

[0084] Figure 5 This is a flowchart illustrating an embodiment of the present disclosure for acquiring a synthesized image.

[0085] Figure 6This is a schematic diagram showing an image before and after image synthesis according to an embodiment of this disclosure.

[0086] Figure 7 This is a flowchart of a camera-coordinated image synthesis method according to an embodiment of the present disclosure.

[0087] Figure 8 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0088] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0089] To address the aforementioned technical problems, this disclosure provides a camera-coordinated image composition method, apparatus, electronic device, and storage medium. The camera-coordinated image composition method is suitable for use as a calling device, including but not limited to smartphones, tablets, or personal computers—devices with processing capabilities and camera modules. Another electronic device acts as the called device, establishing a communication connection with the calling device, such as a Bluetooth or WiFi link, to transmit control commands and image data. It should be noted that the number of called devices can be one or more. For ease of description, subsequent embodiments use a single called device as an example, but this does not constitute a limitation on the present disclosure.

[0090] Taking the electronic device side as an example, this embodiment provides a camera-coordinated image composition method, see [link to relevant documentation]. Figure 1 This includes steps 11 to 14.

[0091] In step 11, the control instructions of the local camera module are synchronized to the called device. The control instructions are used to control the local camera module of the electronic device to acquire local images and to control the remote camera module of the called device to acquire remote images.

[0092] In this step, the electronic device can synchronize the control commands from its local camera module to the called device. Synchronizing the control commands means that upon receiving control commands from the local camera module sent to the electronic device, it forwards these commands to the called device; or, in other words, the control commands can control the local camera module to acquire local images, and control the remote camera module of the called device to acquire remote images.

[0093] Understandably, the approach of the electronic device sending control commands to the camera hardware of the local camera module and then receiving local and meta-images from the camera hardware can be chosen based on the specific scenario and is not limited here. This step describes how to synchronize control commands from the electronic device to the called device.

[0094] In this step, the control commands mentioned above may include a camera activation request, a data stream configuration request, and an image processing capture request. In this scenario, the electronic device synchronizes the control commands from its local camera module to the called device, see [link to relevant documentation]. Figure 2 This includes steps 21 to 23.

[0095] In step 21, in response to detecting a camera activation request sent by a third-party camera application, the camera activation request is synchronized to the remote camera module of the invoked device.

[0096] In this step, the Camera InjectionManager of the Camera Hardware Abstraction Layer (Camera HAL) detects whether a third-party camera application has sent a camera activation request. For example, the Camera InjectionManager can detect whether the open API for enabling the camera has been called by a third-party camera application. When the open API is detected to have been called by a third-party camera application, the Camera InjectionManager can confirm that a camera activation request has been sent by the third-party camera application. At this point, the Camera InjectionManager can synchronize the camera activation request to the remote camera module of the called device.

[0097] Taking an electronic device and the called device as an example, when the CameraInjectionManager detects that a third-party camera application calls the open API to enable the camera, it determines that a camera enable request has been detected. At this time, the CameraInjectionManager can use the AIDL binder mechanism (a mechanism in Android that enables inter-process communication, allowing data exchange or method calls between different processes) to pass the camera enable request to the VirtualCameraService. Then, the VirtualCameraService can send the camera enable request, encoded in JSON-RPC, to the called device via the aforementioned WiFi connection.

[0098] Understandably, after receiving a camera activation request, the virtual camera application (VirtualCameraApp) of the called device will call the open API to attempt to activate the camera of the remote camera module. Thus, by forwarding the camera activation request to the called device in this step, the same request can simultaneously activate both the local and remote cameras, achieving the goal of controlling a remote device from an electronic device.

[0099] In step 22, in response to detecting a data stream configuration request sent by the third-party camera application, the data stream configuration request is synchronized to the remote camera module.

[0100] In this step, the Camera InjectionManager of the Camera Hardware Abstraction Layer (Camera HAL) detects whether a third-party camera application is sending a data stream configuration request. For example, the Camera InjectionManager can detect whether the configureStream API is being called by a third-party camera application. When the configureStream API is detected to be called by a third-party camera application, the Camera InjectionManager can confirm that a data stream configuration request has been sent by the third-party camera application. At this point, the Camera InjectionManager can synchronize the data stream configuration request to the remote camera module of the called device.

[0101] Continuing with the example of an electronic device and the called device using the Android system, when the CameraInjectionManager detects that a third-party camera application is calling the configureStream API to configure the camera data stream, it determines that a data stream configuration request has been sent by the third-party camera application. At this point, the CameraInjectionManager can use the AIDL binder mechanism to pass the data stream configuration request through to the VirtualCameraService. Then, the VirtualCameraService can send the data stream configuration request, encoded in JSON-RPC, to the called device via the aforementioned WiFi connection.

[0102] Understandably, after receiving the data stream configuration request, the virtual camera application VirtualCameraApp of the called device will call the camera data stream interface configureStream API, that is, attempt to request a data stream of the corresponding format and size from the camera of the remote camera module.

[0103] Understandably, the functionality of the camera data stream interface, configureStream API, includes:

[0104] (1) Configure camera data stream: Before taking photos and recording videos, third-party camera applications need to configure the data stream of the remote camera module, such as passing the application layer's Surface container (such as SurfaceView, TextureView for preview, or ImageReader for image processing) to the Camera Server. The Camera Server will wrap the above Surface container into a data stream and pass it to the HAL layer for configuration via HIDL.

[0105] (2) Create a camera capture session CameraCaptureSession: During the creation of CameraCaptureSession, the configureStream operation is called to pass the Surface reference parameter to the Camera Server. This is equivalent to the third-party camera application handing the Surface container to the Camera Server, which converts it into a stream and then passes it to HAL for configuration.

[0106] (3) Simultaneous use of multiple streams: Different streams may require different resolutions or pixel formats. For example, in video recording, one stream is used for previewing, while another stream is encoded and saved to a file; in computational photography, one stream is used for previewing, while another stream is used for face / scene detection.

[0107] (4) Set request template: When taking photos and recording videos, you can create a CameraCaptureRequest containing multiple streams. For example, using the preview capture template TEMPLATE_PREVIEW as a combined stream can optimize for low latency; for high-quality images, you can use TEMPLATE_STILL_CAPTURE; and for a stable frame rate, you can use TEMPLATE_RECORD.

[0108] (5) Set Repeating Request: The setRepeatingRequest method can be used to set a repeating request, and the camera will continuously capture images and fill them into the corresponding Surface container.

[0109] In other words, the ConfigureStream API is used to configure the camera's data stream, create CameraCaptureSession, and manage requests and processing of multiple streams when taking photos and videos, in order to control the camera's output to meet different use cases.

[0110] Understandably, by forwarding the data stream configuration request to the called device in this step, the local camera and the remote camera can be configured simultaneously with a single data stream configuration, which helps to improve configuration efficiency.

[0111] In step 23, in response to detecting an image processing capture request sent by a third-party camera application, the image processing capture request is synchronized to the remote camera module; the camera activation request, the data stream configuration request, and the image processing capture request serve as the control instructions.

[0112] In this step, the Camera InjectionManager of the Camera Hardware Abstraction Layer (Camera HAL) detects whether third-party camera applications are sending image processing capture requests. For example, the Camera InjectionManager can detect whether the processCaptureReques API (for enabling preview or taking a photo) is being called by a third-party camera application. When the processCaptureReques API is detected being called by a third-party camera application, the Camera InjectionManager can confirm that an image processing capture request has been sent by the third-party camera application. At this point, the Camera InjectionManager can synchronize the image processing capture request to the remote camera module of the called device.

[0113] Continuing with the example of an electronic device and the called device using the Android system, when the CameraInjectionManager detects that a third-party camera application is calling the processCaptureRequests API, it determines that an image processing capture request has been sent by the third-party camera application. At this point, the CameraInjectionManager can use the AIDL binder mechanism to pass the image processing capture request through to the VirtualCameraService. Then, the VirtualCameraService can send the image processing capture request, encoded in JSON-RPC, to the called device via the aforementioned WiFi connection.

[0114] Understandably, after receiving an image processing capture request, the virtual camera application VirtualCameraApp of the called device will call the image processing capture request interface processCaptureReques API, that is, attempt to send a request to the remote camera module to start preview or take a picture.

[0115] Understandably, the functionality of the image processing capture request interface, processCaptureRequests API, includes:

[0116] (1) Submitting a capture request: The processCaptureRequest API is used to send a new capture request to the Hardware Abstraction Layer (HAL) to start the image capture process.

[0117] (2) Asynchronous processing: The image processing capture request is processed asynchronously. The capture result is returned by the hardware abstraction layer (HAL) by calling notifyCaptureResult. That is, the processCaptureRequest function does not return the result immediately after sending the image processing capture request, but waits for the hardware abstraction layer (HAL) to complete the processing and then returns the result through another notifyCaptureResult.

[0118] (3) Batch request processing: The processCaptureRequest API supports batch request processing, which sends multiple requests at the same time to maintain the full output frame rate, helping to improve the camera's capture efficiency and response speed.

[0119] (4) Request submission confirmation: Once the request is successfully submitted to the Hardware Abstraction Layer (HAL), processCaptureRequest will mark the request as submitted and update the latest request sent to HAL.

[0120] Understandably, by forwarding the image processing capture request to the called device in this step, the effect of image acquisition by the local camera and the remote camera can be controlled with a single image processing capture request, which helps to improve configuration efficiency.

[0121] As can be seen from steps 21 to 23, by synchronizing the control commands of the electronic device to the called device, the electronic device and the called device can be controlled to perform the same operation, thereby achieving the purpose of using the electronic device to control the called device and realizing the effect of collaborative shooting between the local camera and the remote camera.

[0122] In step 12, the remote image returned by the called device is obtained.

[0123] In this step, after sending control commands, the Camera Hardware Abstraction Layer (HAL) can return the remote image via notifyCaptureResult.

[0124] It should be noted that during the process of acquiring remote images from the called device, the local camera module of the electronic device will also acquire local images according to the control instructions.

[0125] In step 13, the local image and the remote image are combined to obtain a composite image.

[0126] In this step, the Camera Hardware Abstraction Layer (Camera HAL) can acquire local images captured by the local camera module and remote images captured by the remote camera module. Furthermore, the Camera HAL can acquire image configuration formats, including image-in-picture or image-within-picture formats. These image configuration formats can be pre-configured or guided to the user during shooting.

[0127] In one example, the images are arranged side-by-side as follows: Figure 3 As shown. Image side-by-side formatting can include the local image being positioned to the left, right, above, or below the distant image. Figure 3 This example illustrates the effect of a local image being positioned to the left of a distant image.

[0128] In one example, the picture-in-picture format is as follows: Figure 4 As shown. Picture-in-picture format can be either a local image located inside a distant image, or a distant image located inside a local image. Figure 4 This example illustrates the effect of a local image being located inside a distant image.

[0129] It should be noted that, Figure 3 and Figure 4 The example demonstrates the compositing of two images. In some examples, there can be multiple remote images, in which case each compositing operation can combine the local image and multiple remote images into a single composite image.

[0130] In another example, the image configuration format can be configured according to the specific scenario. For example, if the local image contains scenery and the distant image contains people, the people can be superimposed on the scenery to form a composite image of people and scenery superimposed. The corresponding scheme falls within the protection scope of this disclosure.

[0131] In this step, local and remote images can be combined to obtain a composite image. See [link to relevant documentation]. Figure 5This includes steps 51 to 53. In step 51, the image data type requested by the image processing capture request within the control instruction can be obtained. This image data type includes preview type or photo capture type. In step 52, the image configuration format of the composite image is obtained, such as... Figure 3 or Figure 4 As shown. In step 53, the local image and the remote image can be processed according to the image configuration format, such as cropping, layout, and compositing, to obtain a composite image that matches the image data type. In this example, by processing the composite image to match the image data type, it can be ensured that the composite image has the same image data type as the local image, thus meeting the shooting requirements of the electronic device.

[0132] Continuing with the example of an electronic device and the called device using the Android system, the CameraHAL hardware abstraction layer's StreamInjectionSession distributes data to other StreamInjectionSessions based on the image data type: either a preview-type RepeatingCaptureStreamInjectionSession or a photo-type SingleCaptureStreamInjectionSession. The StreamInjectionSession then crops, lays out, and composites the local and remote images according to the image configuration format, resulting in a composite image. Finally, the StreamInjectionSession returns the composite image to the CameraInjectionSession. The CameraInjectionSession then sends the composite image to the CameraInjectionManager.

[0133] In step 14, the synthesized image is displayed on an electronic device.

[0134] In this step, the CameraInjectionManager replaces the local image with the composite image described above, and then passes it sequentially through the CameraService and the Camera2Java API before passing it to the third-party camera application. In this scenario, the third-party camera application can display the composite image, meaning the composite image is displayed on the electronic device's screen, as shown in the image. Figure 6 As shown. See also Figure 6(a) The rear camera 61 of the local camera of the electronic device in the figure can capture local images of local object 62; (b) The rear camera 63 of the remote camera of the called device in the figure can capture remote images of remote object 64; (c) A composite image is displayed in the electronic device in the figure, which includes images of local object 62 and remote object 64. Thus, in this step, by replacing the local image with the composite image on the local image forwarding path, the composite image is uploaded to the third-party camera application according to the local image forwarding path. Image replacement is achieved without the third-party camera application's awareness, achieving camera collaboration between the local and remote cameras, which is beneficial to improving the camera user experience.

[0135] The camera-cooperative image composition method provided in this disclosure is described below with reference to a specific scenario. Figure 7 ,include:

[0136] (1) When a third-party camera application attempts to enable the local camera hardware, it will call the openAPI interface to enable the camera.

[0137] CameraInjectionManager detects open API calls and passes the camera-on request to VirtualCameraService via the AIDL binder mechanism. VirtualCameraService then sends the JSON-RPC encoded camera-on request (or open command) to the called device connected to the electronic device via a WiFi connection.

[0138] After the called device receives the camera activation request, the virtual camera application VirtualCameraApp of the called device will attempt to activate the remote camera module.

[0139] (2) When a third-party camera application requests to configure the camera data stream, it will call the camera data stream interface configureStream API.

[0140] CameraInjectionManager detects calls to the configureStream API; when a call is detected, it passes the data stream configuration request to VirtualCameraService via the AIDL binder mechanism. VirtualCameraService then sends the JSON-RPC encoded data stream configuration request (or configureStream command) to the called device connected to the electronic device via a WiFi link.

[0141] After receiving the configureStream command, the VirtualCameraApp of the called device will attempt to request a data stream of the corresponding format and size from the camera that has been registered.

[0142] (3) When a third-party camera initiates a request to open a preview or take a picture, it will call the image processing capture request interface processCaptureRequests API.

[0143] CameraInjectionManager detects the processCaptureRequest API call and passes the image processing capture request to VirtualCameraService via the AIDL binder mechanism. VirtualCameraService then sends the JSON-RPC encoded image processing capture request (or processCaptureRequest instruction) to the called device connected to the electronic device via a WiFi link.

[0144] After receiving the processCaptureRequest instruction, the VirtualCameraApp on the called device will send a request to the remote camera to start a preview or take a picture.

[0145] (4) After the preview or photo capture request is completed, Camera HAL will report the remote image via notifyCaptureResult; VirtualCameraApp will send the remote image to the VirtualCameraService of the electronic device via WiFi connection; VirtualCameraService will pass the received remote image to VirtualCamera via AIDL binder mechanism. VirtualCamera will then pass the remote image to CameraInjectionSession.

[0146] Furthermore, CameraInjectionManager will send the acquired local image to CameraInjectionSession.

[0147] (5) CameraInjectionSession acquires both local and remote images simultaneously.

[0148] (6) CameraInjectionSession will distribute the data to the corresponding StreamInjectionSession's RepeatingCaptureStreamInjectionSession or SingleCaptureStreamInjectionSession based on the image data type (preview or photo).

[0149] (7) StreamInjectionSession will crop, lay out and composite the local image and the remote image according to the image configuration format to obtain the composite image.

[0150] (8) StreamInjectionSession returns the composite image to CameraInjectionSession.

[0151] (9) CameraInjectionSession forwards the composite image to CameraInjectionManager.

[0152] (10) CameraInjectionManager replaces the local image in step (4) with the composite image received in step (9).

[0153] (11) Display the composite image in a third-party camera application. If it is a preview type, it will be displayed in the preview interface of the third-party camera application; if it is a photo type, the third-party camera application will display the received photo image as the composite image mentioned above.

[0154] In this way, the images of the electronic device and the called device can be combined, which maximizes the utilization of multi-camera hardware resources and improves the traditional camera collaboration function with multiple images and multiple perspectives. It can optimize the user experience in user scenarios such as online teaching or live video streaming.

[0155] Based on the camera cooperative image composition method provided in the embodiments of this disclosure, the embodiments of this disclosure also provide a camera cooperative image composition apparatus, including:

[0156] The camera hardware abstraction layer is used to synchronize control commands from the local camera module to the called device. The control commands are used to control the local camera module of the electronic device to acquire local images and to control the remote camera module of the called device to acquire remote images.

[0157] The camera hardware abstraction layer is used to obtain the remote image returned by the called device;

[0158] The camera hardware abstraction layer is used to synthesize the local image and the remote image to obtain a synthesized image;

[0159] The camera hardware abstraction layer is used to display the synthesized image on an electronic device.

[0160] In one embodiment, the camera hardware abstraction layer synchronizes control commands from the local camera module to the invoked device, including:

[0161] The camera hardware abstraction layer is used to respond to a camera activation request sent by a third-party camera application and synchronize the camera activation request to the remote camera module of the invoked device.

[0162] The camera hardware abstraction layer is used to synchronize the data stream configuration request to the remote camera module in response to detecting the data stream configuration request sent by the third-party camera application.

[0163] The camera hardware abstraction layer is used to synchronize the image processing and capture request to the remote camera module in response to the detection of an image processing and capture request sent by a third-party camera application.

[0164] In one embodiment, the camera hardware abstraction layer synchronizes the camera activation request to the remote camera module of the invoked device, including:

[0165] When the camera injection manager of the electronic device detects that the third-party camera application calls the camera start interface, it determines that it has obtained the camera start request.

[0166] The camera activation request is passed through to the virtual camera service;

[0167] The virtual camera service sends the camera activation request to the invoked device; the camera activation request is used to instruct the virtual camera application of the invoked device to attempt to activate the remote camera module.

[0168] In one embodiment, the camera hardware abstraction layer synchronizes the data stream configuration request to the remote camera module, including:

[0169] When the camera injection manager of the electronic device detects that the third-party camera application requests to call the camera data stream interface, it determines that it has obtained the data stream configuration request;

[0170] The data stream configuration request is passed through to the virtual camera service;

[0171] The virtual camera service sends the data stream configuration request to the invoked device; the data stream configuration request is used to instruct the virtual camera application of the invoked device to attempt to request the corresponding data stream from the already activated camera.

[0172] In one embodiment, the camera hardware abstraction layer synchronizes the image processing capture request to the remote camera module, including:

[0173] When the camera injection manager of the electronic device detects that the third-party camera application requests to call the interface to initiate a preview or image processing capture request, it determines that the image processing capture request has been obtained.

[0174] The image processing capture request is passed through to the virtual camera service;

[0175] The virtual camera service sends the image processing and capture request to the invoked device; the image processing and capture request is used to instruct the virtual camera application of the invoked device to send a request to start previewing or take a picture.

[0176] In one embodiment, the camera hardware abstraction layer synthesizes the local image and the remote image to obtain a synthesized image, including:

[0177] Obtain the image data type requested by the image processing capture request within the control command, wherein the image data type includes preview type or photo type;

[0178] The image configuration format of the synthesized image is obtained, including an image side-by-side format or a picture-in-picture format;

[0179] The local image and the remote image are processed according to the image configuration format to obtain a composite image that matches the image data type.

[0180] In one embodiment, the camera hardware abstraction layer acquires the remote image returned by the called device, including:

[0181] The virtual camera service within the camera hardware abstraction layer of the electronic device acquires the remote image fed back by the invoked device;

[0182] The virtual camera service sends the remote image to the virtual camera;

[0183] The virtual camera sends the remote image to the camera injection session to obtain the remote image captured by the remote camera module.

[0184] In one embodiment, the camera hardware abstraction layer displays the synthesized image on an electronic device, including:

[0185] The stream injection session of the camera hardware abstraction layer within the electronic device returns the synthesized image to the camera injection session.

[0186] The camera injection session forwards the composite image to the camera injection manager;

[0187] The camera injection manager replaces the local image with the synthesized image;

[0188] The camera injection manager passes the synthesized image through to a third-party camera application.

[0189] It should be noted that the apparatus shown in this embodiment matches the content of the method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0190] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a smartphone, computer, digital broadcasting terminal, tablet device, medical device, fitness equipment, personal digital assistant, etc. This electronic device can implement the aforementioned camera-coordinated image composition method. It is understood that this electronic device can serve as either a calling device or a called device in implementing the aforementioned camera-coordinated image composition method.

[0191] Reference Figure 8 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, communication component 816, and image acquisition component 818.

[0192] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute computer programs. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802. In one example, the processing component may include a processor for performing the methods described above.

[0193] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include computer programs for any application or method operating on electronic device 800, contact data, phone book data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0194] Power supply assembly 806 provides power to various components of electronic device 800. Power supply assembly 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800. Power supply assembly 806 may include a power chip, and a controller may communicate with the power chip to control the power chip to turn on or off a first switching device, thereby enabling or disabling battery power to the circuit board.

[0195] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and target object. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input information from the target object. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0196] Audio component 810 is configured to output and / or input audio file information. For example, audio component 810 includes a microphone (MIC) configured to receive external audio file information when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio file information may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio file information.

[0197] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc.

[0198] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display screen and keypad of electronic device 800, changes in the position of electronic device 800 or a component, the presence or absence of a target object in contact with electronic device 800, the orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800.

[0199] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast information or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0200] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital information processors (DSPs), digital information processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0201] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the method as described above.

[0202] In an exemplary embodiment, a non-transitory computer-readable storage medium is also provided, which enables the implementation of the method described above when an executable computer program in the storage medium is executed by a processor.

[0203] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and the practice of the disclosed solutions. This disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this disclosure is indicated by the claims.

[0204] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for camera-coordinated image composition, characterized in that, The method includes: The control commands of the local camera module are synchronized to the called device. The control commands are used to control the local camera module of the electronic device to acquire local images and to control the remote camera module of the called device to acquire remote images. Obtain the remote image returned by the called device; The local image and the remote image are combined to obtain a composite image; The composite image is displayed on an electronic device.

2. The method according to claim 1, characterized in that, Synchronize the control commands of the local camera module to the called device, including: In response to detecting a camera activation request sent by a third-party camera application, the camera activation request is synchronized to the remote camera module of the invoked device. In response to detecting a data stream configuration request sent by the third-party camera application, the data stream configuration request is synchronized to the remote camera module; In response to detecting an image processing capture request sent by a third-party camera application, the image processing capture request is synchronized to the remote camera module.

3. The method according to claim 2, characterized in that, Synchronizing the camera activation request to the remote camera module of the invoked device includes: When the camera injection manager of the electronic device detects that the third-party camera application calls the camera start interface, it determines that it has obtained the camera start request. The camera activation request is passed through to the virtual camera service; The virtual camera service sends the camera activation request to the invoked device; the camera activation request is used to instruct the virtual camera application of the invoked device to attempt to activate the remote camera module.

4. The method according to claim 2, characterized in that, Synchronizing the data stream configuration request to the remote camera module includes: When the camera injection manager of the electronic device detects that the third-party camera application requests to call the camera data stream interface, it determines that it has obtained the data stream configuration request; The data stream configuration request is passed through to the virtual camera service; The virtual camera service sends the data stream configuration request to the invoked device; the data stream configuration request is used to instruct the virtual camera application of the invoked device to attempt to request the corresponding data stream from the already activated camera.

5. The method according to claim 2, characterized in that, Synchronizing the image processing capture request to the remote camera module includes: When the camera injection manager of the electronic device detects that the third-party camera application requests to call the interface to initiate a preview or image processing capture request, it determines that the image processing capture request has been obtained. The image processing capture request is passed through to the virtual camera service; The virtual camera service sends the image processing and capture request to the invoked device; the image processing and capture request is used to instruct the virtual camera application of the invoked device to send a request to start previewing or take a picture.

6. The method according to claim 1, characterized in that, The local image and the remote image are combined to obtain a synthesized image, including: Obtain the image data type requested by the image processing capture request within the control command, wherein the image data type includes preview type or photo type; The image configuration format of the synthesized image is obtained, including an image side-by-side format or a picture-in-picture format; The local image and the remote image are processed according to the image configuration format to obtain a composite image that matches the image data type.

7. The method according to claim 1, characterized in that, Obtaining the remote image returned by the called device includes: The virtual camera service within the camera hardware abstraction layer of the electronic device acquires the remote image fed back by the invoked device; The virtual camera service sends the remote image to the virtual camera; The virtual camera sends the remote image to the camera injection session to obtain the remote image captured by the remote camera module.

8. The method according to claim 1, characterized in that, Displaying the composite image on an electronic device includes: The stream injection session of the camera hardware abstraction layer within the electronic device returns the synthesized image to the camera injection session. The camera injection session forwards the composite image to the camera injection manager; The camera injection manager replaces the local image with the synthesized image; The camera injection manager passes the synthesized image through to a third-party camera application.

9. A camera-coordinated image synthesis device, characterized in that, The device includes: The camera hardware abstraction layer is used to synchronize control commands from the local camera module to the called device. The control commands are used to control the local camera module of the electronic device to acquire local images and to control the remote camera module of the called device to acquire remote images. The camera hardware abstraction layer is used to obtain the remote image returned by the called device; The camera hardware abstraction layer is used to synthesize the local image and the remote image to obtain a synthesized image; The camera hardware abstraction layer is used to display the synthesized image on an electronic device.

10. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs that can be executed by the processor; The processor is configured to execute a computer program in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computational program / instruction is executed by the processor, it implements the method described in any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium, characterized in that, When the executable computer program in the storage medium is executed by a processor, it can implement the method as described in any one of claims 1 to 8.

Images