Camera and Image Processing Method Based on Android System

Abstract

The present application discloses a camera and an image processing method based on the Android system. The method includes: an image sensor collects image data to be processed, and a data processing module preprocesses the image data to be processed to obtain camera image data; a component for synthesizing an image receives a first interface image obtained by a display module and the camera image data, and performs superposition processing to obtain image data to be output, and transmits the image data to be output to a corresponding display terminal for display through a video interface; the component for synthesizing an image receives a second interface image obtained by a virtual display module and the camera image data, and performs superposition processing to obtain image data to be output; the image data to be output is transmitted to a corresponding display terminal for display through a peripheral interface, and the interface image generated by grabbing a surface through the virtual display module is used for image superposition as the video source of the camera, enhancing the portability of the system and reducing the complexity of visual image processing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a camera and an image processing method based on the Android system. Background Art

[0002] In traditional cameras, image processing is carried out through the linux system. During the processing, the linux system interface is called to process images. If a graphical interface needs to be displayed during the processing, a large amount of code development and the use of a large number of open-source software are required, and the processing process is complex.

[0003] Content of the Application

[0004] In view of this, embodiments of this application provide a camera and an image processing method based on the Android system, aiming to solve the technical problem of complex image processing.

[0005] This application provides a camera. The camera runs the Android system. The physical link layer of the Android system includes an image sensor, a video interface, and a peripheral interface. The hardware abstraction layer of the Android system includes a data processing module and multiple components for synthesizing images. The application layer of the Android system includes two selectable surfaces for displaying interface images. One surface obtains a first interface image through a display module, and the other surface obtains a second interface image through a virtual display module. The display module and the virtual display module respectively correspond one-to-one with the components for synthesizing images. The display module transmits the first interface image to the corresponding component for synthesizing images, and the virtual display module transmits the second interface image to the corresponding component for synthesizing images;

[0006] The image sensor is used to collect image data to be processed and transmit the image data to be processed to the data processing module through the kernel layer of the Android system;

[0007] The data processing module is used to preprocess the image data to be processed to obtain camera image data, and transmit the camera image data to the components for synthesizing images corresponding to the display module and the virtual display module respectively;

[0008] The component for synthesizing images corresponding to the display module is used to receive the first interface image obtained by the display module and the camera image data, perform superimposition processing on the camera image data and the first interface image to obtain image data to be output, and transmit the image data to be output to the corresponding display terminal through the video interface for display;

[0009] The component of the synthetic image corresponding to the virtual display module is used to receive the second interface image obtained by the virtual display module and the camera image data, and superimpose the camera image data and the second interface image to obtain the image data to be output, and transmit the image data to be output to the corresponding display terminal through the peripheral interface for display.

[0010] In a possible implementation of the present application, the physical link layer of the Android system further includes a virtual interface image buffer;

[0011] The virtual interface image cache area is used to obtain the cache space allocated by the preset memory address space, and cache the second interface image obtained by the virtual display module through the virtual interface image cache pointer to obtain a cached interface image, and update the cache space after the caching is completed.

[0012] In a possible implementation of the present application, the physical link layer of the Android system further includes a camera image data buffer area;

[0013] The camera image data buffer area is used to obtain a camera image caching rule and a refresh rate of a display terminal corresponding to the peripheral interface, calculate and generate a generation time of each frame of camera image data according to the refresh rate, and perform multi-frame caching of the camera image data through a camera image cache pointer based on the generation time of each frame of the camera image data and the camera image caching rule;

[0014] The virtual interface image cache area is also used to obtain the interface image cache rules and the refresh rate of the display terminal corresponding to the peripheral interface, calculate the generation time of each frame of the second interface image through the refresh rate, and cache multiple frames of the second interface image through the virtual interface image cache pointer based on the generation time of each frame of the second interface image and the interface image cache rules.

[0015] In a possible implementation of the present application, the component of the synthetic image corresponding to the virtual display module is further used to calculate the cache duration of the cached virtual interface image based on the interface image cache rule and the generation duration of each frame of the second interface image, and when the waiting time satisfies the cache duration of the cached virtual interface image, obtain the initial virtual interface image from the virtual interface image cache area through the virtual interface image cache pointer, obtain the virtual interface image cache pointer information corresponding to the initial virtual interface image, and determine whether the initial virtual interface image is complete based on the virtual interface image cache pointer information. When the initial virtual interface image is complete, use the initial virtual interface image as the second interface image;

[0016] The component of the composite image corresponding to the virtual display module is further configured to calculate the caching duration for caching the camera image data based on the camera image caching rule and the generation duration of each frame of the camera image data. When the waiting duration meets the caching duration for caching the camera image data, the initial camera image data is obtained from the camera image data buffer through the camera image caching pointer, and the camera image caching pointer information corresponding to the initial camera image data is obtained. Then, it is determined whether the initial camera image data is complete based on the camera image caching pointer information. When the initial camera image data is complete, the initial camera image data is used as the camera image data;

[0017] The component of the composite image corresponding to the virtual display module is further configured to perform an overlay process on the camera image data and the second interface image to obtain the image data to be output.

[0018] In a possible implementation manner of the present application, the hardware abstraction layer of the Android system further includes: an encoder;

[0019] The encoder is configured to receive the image data to be output transmitted by the composite component corresponding to the virtual display module, parse the image data to be output to determine the data format of the image data to be output. When the data format of the image data to be output is the preset encoding format, perform encoding processing on the image data to be output in the preset encoding format to obtain the encoded image data to be output, and transmit the encoded image data to the corresponding display terminal through the peripheral interface for display.

[0020] The embodiment of the present application further provides an image processing method based on the Android system. The image processing method based on the Android system is applied to the camera as described above. The method includes:

[0021] The image sensor collects the image data to be processed and transmits the image data to be processed to the data processing module through the kernel layer of the Android system;

[0022] The data processing module preprocesses the image data to be processed to obtain the camera image data, and transmits the camera image data to the composite image component corresponding to the display module and the virtual display module respectively;

[0023] The composite image component corresponding to the display module receives the first interface image obtained by the display module and the camera image data, and performs an overlay process on the camera image data and the first interface image to obtain the image data to be output;

[0024] Transmit the image data to be output to the corresponding display terminal through the video interface for display;

[0025] The component of the composite image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superimposition processing on the camera image data and the second interface image to obtain the image data to be output;

[0026] Transmit the image data to be output to the corresponding display terminal through the peripheral interface for display.

[0027] In a possible implementation manner of the present application, before the component of the composite image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superimposition processing on the camera image data and the second interface image to obtain the image data to be output, it further includes:

[0028] Generate a virtual interface image buffer creation instruction;

[0029] Allocate a buffer space in the preset memory address space according to the virtual interface image buffer creation instruction to establish a virtual interface image buffer;

[0030] Cache the second interface image obtained by the virtual display module into the virtual interface image buffer through the virtual interface image buffer pointer;

[0031] After the caching is completed, update the buffer space.

[0032] In a possible implementation manner of the present application, the component of the composite image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superimposition processing on the camera image data and the second interface image to obtain the image data to be output, including:

[0033] Obtain the refresh rate of the display terminal corresponding to the peripheral interface, the camera image caching rule, and the interface image caching rule;

[0034] Calculate the generation duration of each frame of the camera image data and the generation duration of each frame of the second interface image through the refresh rate;

[0035] Calculate the caching duration of the camera image data by the generation duration of each frame of the camera image data and the camera image caching rule;

[0036] Calculate the caching duration of the virtual interface image by the generation duration of each frame of the second interface image and the interface image caching rule;

[0037] When the waiting duration meets the caching duration for caching the virtual interface image, obtain the initial virtual interface image through the virtual interface image caching pointer, and when the waiting duration meets the caching duration for caching the camera image data, obtain the initial camera image data through the camera image caching pointer;

[0038] Obtain the virtual interface image caching pointer information corresponding to the initial virtual interface image and the camera image caching pointer information corresponding to the initial camera image data;

[0039] Judge whether the initial virtual interface image is complete according to the virtual interface image caching pointer information, and judge whether the initial camera image data is complete according to the camera image caching pointer information;

[0040] When the initial virtual interface image is complete, use the initial virtual interface image as the second interface image;

[0041] When the initial camera image data is complete, use the initial camera image data as the camera image data;

[0042] Perform superposition processing on the second interface image and the camera image data to obtain the image data to be output.

[0043] In a possible implementation manner of the present application, before transmitting the image data to be output to the corresponding display terminal for display through the peripheral interface, it further includes:

[0044] Obtain the UVC message between the peripheral interface and the corresponding display terminal;

[0045] Parse the UVC message to determine the transmission protocol;

[0046] Generate a parameter adjustment instruction through the transmission protocol;

[0047] Adjust the parameters during image sensor acquisition through the parameter adjustment instruction.

[0048] In a possible implementation manner of the present application, the transmitting the image data to be output to the corresponding display terminal for display through the peripheral interface includes:

[0049] Parse the image data to be output to determine the data format of the image data to be output;

[0050] When the data format of the image data to be output is the YUV format, transmit the YUV-format image data to be output to the corresponding display terminal for display through the peripheral interface;

[0051] When the data format of the to-be-output image data is a preset encoding format, generate an encoding control instruction through the transmission protocol;

[0052] Perform encoding processing on the to-be-output image data in the preset encoding format through the encoding control instruction to obtain the encoded to-be-output image data;

[0053] Transmit the encoded to-be-output image data to a corresponding display terminal through the peripheral interface for display.

[0054] This application also provides an image processing device based on the Android system. The image processing device based on the Android system includes: a memory, a processor, and a program of an image processing method based on the Android system stored on the memory and executable on the processor. When the program of the image processing method based on the Android system is executed by the processor, the steps of the image processing method based on the Android system as described above can be implemented.

[0055] To achieve the above object, a storage medium is also proposed. An image processing method program based on the Android system is stored on the storage medium. When the program of the image processing method based on the Android system is executed by the processor, the steps of the image processing method based on the Android system as described above are implemented.

[0056] The present application provides a camera and an image processing method based on the Android system. Compared with the current image processing through the Linux system, where the process of displaying the processed image interface is relatively cumbersome and complex and the cost is high, in the present application, an image sensor collects image data to be processed; and transmits the image data to be processed to a data processing module through the kernel layer of the Android system, so as to perform preprocessing through the data processing module to obtain camera image data; a component for synthesizing an image corresponding to a display module receives a first interface image obtained by the display module and the camera image data, and performs superposition processing on the camera image data and the first interface image to obtain image data to be output. The image data to be output can be transmitted to a corresponding display terminal through a video interface for display. A component for synthesizing an image corresponding to a virtual display module receives a second interface image obtained by the virtual display module and the camera image data, and performs superposition processing on the camera image data and the second interface image to obtain image data to be output, and transmits the image data to be output to a corresponding display terminal through a peripheral interface for display. An interface image based on the Android system can be obtained quickly, facilitating the design, development, and implementation of the interface image, providing a better human-computer interaction interface, and being able to capture the interface image generated by a surface through the virtual display module, and performing superposition on the interface image and the camera image data through the component for synthesizing an image to obtain the image data to be output of the composite interface, which is used as the video source of the camera, enhancing the portability of the system, and displaying the image processing interface in real time during the superposition processing, with a simple operation process and reduced complexity of visual image processing. Description of the Drawings

[0057] Figure 1 It is a schematic diagram of the architecture of the camera of the present application;

[0058] Figure 2 It is a schematic diagram of the process of the first embodiment of the image processing method based on the Android system of the present application;

[0059] Figure 3 It is a schematic diagram of the process of processing the image data to be processed involved in the image processing method based on the Android system of the present application;

[0060] Figure 4 It is a schematic diagram of the scenario of synthesizing an image involved in the image processing method based on the Android system of the present application;

[0061] Figure 5 It is a schematic diagram of the device structure of the hardware operating environment involved in the solution of the embodiment of the present application. Detailed Embodiment

[0062] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The embodiment of the present application provides an image processing method based on the Android system. In the first embodiment of the image processing method based on the Android system of the present application, with reference to Figure 2 , the image processing method based on the Android system includes:

[0064] Step S10: The image sensor collects the image data to be processed and transmits the image data to be processed to the data processing module through the kernel layer of the Android system;

[0065] Step S20: The data processing module preprocesses the image data to be processed to obtain camera image data, and transmits the camera image data to the components for synthesizing images corresponding to the display module and the virtual display module respectively;

[0066] Step S30: The component for synthesizing images corresponding to the display module receives the first interface image obtained by the display module and the camera image data, and superimposes the camera image data and the first interface image to obtain the image data to be output;

[0067] Step S40: Transmit the image data to be output to the corresponding display terminal through the video interface for display;

[0068] Step S50: The component for synthesizing images corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and superimposes the camera image data and the second interface image to obtain the image data to be output;

[0069] Step S60: Transmit the image data to be output to the corresponding display terminal through the peripheral interface for display.

[0070] In this embodiment, it should be noted that:

[0071] The image sensor can be a CCD (Charge-coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0072] The image sensor is located at the physical link layer of the Android system. The image sensor collects the image data to be processed and sends the image data to be processed to the data processing module in the hardware abstraction layer of the Android system through the kernel layer of the Android system for preprocessing. When the data processing module in the hardware abstraction layer of the Android system preprocesses the image to be processed, it can display a graphical interface, so as to obtain camera image data, and the display of the graphical interface can be realized without a cumbersome image processing process.

[0073] In this embodiment, the research and development background is as follows:

[0074] For traditional cameras to process images, one implementation method is to embed a real-time operating system inside the camera to process the images. Through the SDK (Software Development Kit) interface provided by the chip manufacturer, the product manufacturer uses the SDK interface to develop corresponding functions to process the images. However, since it depends on the SDK provided by the chip manufacturer, if the provided SDK is not stable, it will cause situations such as camera disconnection and abnormal image processing, resulting in poor operability of the platform. Another implementation method is to embed a linux system inside the camera. The linux operating system provides many system interfaces and open-source libraries. The chip manufacturer manufactures corresponding chips to adapt to the linux operating system, and the product manufacturer can call the system interfaces of the linux operating system to process the images, and can directly call the system interfaces of the linux operating system for processing, with strong platform portability. However, when performing image processing, if a graphical interface needs to be displayed, a large amount of code development and the use of a large number of open-source software are required, which takes a long time and the processing process is complex.

[0075] Currently, if a graphical interface is to be displayed when processing images, it takes a lot of time and cost, resulting in difficulty in realizing the display of the graphical interface when processing images. That is to say, in the prior art, a large amount of code development and a large number of open-source software are required to display the graphical interface, and the processing process is complex.

[0076] In this embodiment, it is intended that the component corresponding to the composite image of the display module receives the first interface image obtained by the display module and the camera image data transmitted by the data processing module, and superimposes the first interface image and the camera image data to obtain the to-be-output image data in which the camera image and the first interface image are combined. And the component corresponding to the composite image of the virtual reality module receives the second interface image obtained by the virtual display module and the camera image data transmitted by the data processing module, and superimposes the second interface image and the camera image data to obtain the to-be-output image data in which the camera image and the second interface image are combined. The first interface image refers to the interface image generated by the surface and transmitted through the display module display, and the second interface image refers to the interface image generated by the surface and transmitted through the virtual reality module Virtual display. The component of the composite image displays the image processing interface in real time during the superimposing process, with a simple operation process, facilitating the design, development, and implementation of the interface image, and reducing the complexity of visual image processing. And the virtual display module can capture the interface image generated by the surface, and the component of the composite image superimposes the interface image and the camera image data to obtain the to-be-output image of the composite interface, which is used as the video source of the camera, enhancing the portability of the system.

[0077] The specific steps are as follows:

[0078] Step S10, the image sensor collects the to-be-processed image data and transmits the to-be-processed image data to the data processing module through the kernel layer of the Android system;

[0079] As an example, the image processing method based on the Android system can be applied to a camera on which the Android system runs.

[0080] As an example, the physical link layer of the Android system includes an image sensor, a video interface, and a peripheral interface. The hardware abstraction layer of the Android system includes a data processing module and multiple components of composite images. The application layer of the Android system includes at least two surfaces for displaying interface images. One surface obtains the first interface image through the display module, and the other surface obtains the second interface image through the virtual display module. The display module and the virtual display module respectively correspond one-to-one with the components of the composite image. The display module transmits the first interface image to the corresponding component of the composite image, and the virtual display module transmits the second interface image to the corresponding component of the composite image, as shown in Figure 1 shown.

[0081] Among them, the image sensor can be a CCD image sensor, a CMOS image sensor, etc. The image sensor can use the photoelectric conversion function of optoelectronic devices to convert the optical image on the photosensitive surface into an electrical signal that is in a corresponding proportional relationship with the optical image, so as to obtain the image data to be processed. The image sensor transmits the image data to be processed to the data processing module through the kernel layer of the Android system. The data processing module can preprocess the image data to be processed.

[0082] As an example, referring to Figure 1 , the Android system runs on a camera, and the image sensor in the physical link layer of the Android system collects the image data to be processed. After the image sensor captures an image, it obtains the image data to be processed and sends the image data to be processed to the data processing module in the hardware abstraction layer of the Android system through the kernel layer of the Android system. The data processing module preprocesses the image data to be processed to obtain camera image data. The data processing module transmits the camera image data to the components that synthesize images corresponding to the display module and the virtual reality module. The components that synthesize images also receive the first interface image obtained by the display module and the second interface image obtained by the virtual display module. The components that synthesize images corresponding to the display module superimpose the first interface image and the camera image data to obtain the image data to be output, and transmit the image data to be output to the corresponding display terminal through the video interface. The components that synthesize images corresponding to the virtual display module superimpose the second interface image and the camera image data to obtain the image data to be output, and transmit the image data to be output to the corresponding display terminal through the peripheral interface for display.

[0083] As an example, the data processing module can be HAL (Hardware Abstraction Layer), and HAL preprocesses the image data to be processed collected by the image sensor.

[0084] Step S20, the data processing module preprocesses the image data to be processed to obtain camera image data, and transmits the camera image data to the components that synthesize images corresponding to the display module and the virtual display module respectively;

[0085] In this embodiment, preprocessing the image to be processed includes performing dark current subtraction (removing the bottom electrical current noise), linearization, defective pixel removal, denoising, color space conversion, and color enhancement processing on the image to be processed, and also includes using intelligent correlation technologies, such as face detection, face tracking, or key point detection technologies, to perform beauty enhancement, whitening, and background replacement on the image to be processed collected by the image sensor. This embodiment does not limit this. The camera image data refers to the image data after preprocessing the image data to be processed.

[0086] As an example, the image data includes picture data, video stream data, etc.

[0087] As an example, the hardware abstraction layer of the Android system includes a data processing module, and the data processing module can be the HAL (Hardware Abstraction Layer), such as Figure 3 shown Figure 3 is a schematic flowchart for processing the image data to be processed. The data processing module includes an ISP (Image Signal Processor) parameter control module, a 3A algorithm module, and a video acquisition module. The 3A algorithm in the 3A algorithm module includes an AF autofocus algorithm, an AE automatic exposure algorithm, and an AWB automatic white balance algorithm, and the parameters are dynamically configured to the ISP parameter control module through the 3A algorithm. The ISP parameter control module also receives statically configured parameters, and controls the parameters when the image sensor acquires an image through the ISP parameter control module, such as resolution, signal-to-noise ratio, noise, sensitivity, etc. The ISP hardware module receives the image to be processed acquired by the image sensor and preprocesses the image to be processed. The ISP hardware module internally includes multiple image algorithm processing modules, such as a linearization algorithm processing module, a denoising algorithm processing module, a data conversion module, a color space conversion module, and a color processing module, etc. The image to be processed is processed through the ISP hardware module to obtain camera image data. The ISP transmits the camera image data to the video acquisition module, and the video acquisition module transmits the received camera image data to the components for synthesizing images corresponding to the display module and the components for synthesizing images corresponding to the virtual display module respectively.

[0088] Step S30, the component for synthesizing the image corresponding to the display module receives the first interface image obtained by the display module and the camera image data, and performs superposition processing on the camera image data and the first interface image to obtain the image data to be output;

[0089] In this embodiment, the first interface image is an interface image generated by the application layer of the Android system through the surface as a graphical interface, and the first interface image is transmitted to the component for synthesizing the image corresponding to the display module through the display module dispaly interface. Since the App application layer displays the content to be displayed on the screen, the Android system application layer is responsible for initiating a Surface creation request and creating a Surface, thereby generating the first interface image.

[0090] As an example, the first interface image generated by the surface can be transmitted to the corresponding component for synthesizing the image through the display module display interface.

[0091] As an example, when the display terminal is a monitor or an HDMI input device with an HDMI interface, since the camera with an HDMI interface has a display module display interface, the surface can be directly obtained through the display module display interface as the first interface image generated by the graphical interface display, and the obtained first interface image is transmitted to the component for synthesizing the image corresponding to the display module.

[0092] In this embodiment, the component for synthesizing the image can be HWC (hwcomposer, hardware composition abstraction layer). HWC is located in the hardware abstraction layer of the Android system and can perform image synthesis and display through HWC.

[0093] As an example, the component for synthesizing the image receives the first interface image generated by the surface through the display module display interface and the camera image data sent by the data processing module, superimposes the camera image data and the first interface image, and obtains the superimposed image data, that is, the image data to be output.

[0094] Step S40, transmitting the image data to be output to the corresponding display terminal through the video interface for display;

[0095] As an example, the physical link layer of the Android system includes a video interface. The video interface is an HDMI interface. The HDMI interface is connected to the corresponding display terminal. The display terminal connected to the video interface is a monitor or an HDMI input device with an HDMI interface. The component HWC for synthesizing the image can be connected to video interfaces such as HDMI through the video driver. The video interface is connected to the corresponding display terminal, so as to receive the image data to be output and display it on the display terminal.

[0096] Step S50, the component for synthesizing the image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and superimposes the camera image data and the second interface image to obtain the image data to be output;

[0097] As an example, the component for synthesizing the image corresponding to the virtual display module receives the second interface image obtained through the surface by the Virtual display interface of the virtual display module and the camera image data transmitted by the data processing module, superimposes the camera image data and the second interface image, and obtains the superimposed image data, that is, the image data to be output.

[0098] Step S60, transmitting the image data to be output to the corresponding display terminal through the peripheral interface for display.

[0099] As an example, the physical link layer of the Android system further includes a peripheral interface. The peripheral interface is a USB interface, which is connected to a corresponding display terminal, and can transmit the image data to be output to the corresponding display terminal through the peripheral interface. The display terminal connected to the peripheral interface is a display terminal with a USB port. When the display terminal is a device of the USB interface type, the component HWC for synthesizing images can be connected to the USB interface through the UVC gadget driver, and the USB interface is connected to the corresponding display terminal, so as to receive the image data to be output and display it on the display terminal.

[0100] As an example, the physical link layer of the Android system further includes a virtual interface image buffer; the virtual interface image buffer is used to obtain a buffer space allocated from a preset memory address space, and cache the second interface image obtained by the virtual display module through a virtual interface image buffer pointer to obtain a cached interface image, and update the buffer space after the caching is completed.

[0101] As an example, the virtual interface image buffer is located in the physical link layer of the Android system and can be used to cache the second interface image obtained by the virtual display module. The preset memory address space can be a physical memory address space or a virtual memory address space. This embodiment does not limit this, and this embodiment takes the physical memory address space as an example for illustration. By allocating a buffer space for caching the second interface image in the physical memory address space, and caching the second interface image obtained by the virtual display module through a virtual interface image buffer pointer, caching the image through the virtual interface image buffer pointer, and after the caching is completed, sending a completion message to the virtual interface image buffer to enable the virtual interface image buffer to update the size of the buffer space for subsequent virtual interface image caching.

[0102] As an example, the physical link layer of the Android system further includes a camera image data buffer; the camera image data buffer is used to obtain a camera image caching rule and the refresh rate of the display terminal corresponding to the peripheral interface, calculate the generation duration of each frame of the camera image data through the refresh rate, and perform multi-frame caching of the camera image data through a camera image buffer pointer based on the generation duration of each frame of the camera image data and the camera image caching rule;

[0103] The virtual interface image buffer is further used to obtain an interface image caching rule and the refresh rate of the display terminal corresponding to the peripheral interface, calculate the generation duration of each frame of the second interface image through the refresh rate, and perform multi-frame caching of the second interface image through a virtual interface image buffer pointer based on the generation duration of each frame of the second interface image and the interface image caching rule.

[0104] In this embodiment, the camera image data buffer is used to buffer the camera image data processed by the data processing module. The camera image data buffer is set in the physical link layer of the Android system and can receive the camera image data transmitted by the data processing module. The camera image caching rule can be set according to user needs. To solve the system jitter problem, the camera image data buffer can set multi-frame buffering for the camera image data. The number of buffered frames can be set according to requirements, such as 4-frame buffering, 6-frame buffering, etc. In this embodiment, 4-frame buffering is taken as an example for illustration. The refresh rate of the display terminal corresponding to the peripheral interface is the refresh rate of the display terminal of the USB interface type, such as 50HZ, 100HZ, etc. In this embodiment, 50HZ is taken as an example for illustration. After determining the refresh rate, the generation duration of each frame of the camera image data can be calculated based on the refresh rate. For example, if the refresh rate is 50HZ, the generation duration of each frame of the camera image data is 20ms, and then a frame of data is generated every 20ms. Thus, multi-frame buffering of the camera image data can be performed through the camera image cache pointer according to the generation duration of each frame and the camera image caching rule. For example, if a frame of data is generated every 20ms, then a frame of data is written through the camera image cache pointer every 20ms, and a total of 4 frames of data are buffered.

[0105] As an example, the virtual interface image buffer can obtain the set interface image caching rule. To solve the system jitter problem, the virtual interface image buffer can set multi-frame buffering for the second interface image. The number of buffered frames can be set according to requirements, such as 4-frame buffering, 6-frame buffering, etc. In this embodiment, 4-frame buffering is taken as an example for illustration. For example, if the refresh rate of the display terminal corresponding to the peripheral interface is 50HZ, then the generation duration of each frame of the second interface image is calculated as 20ms based on the refresh rate, and a frame of data is generated every 20ms. Thus, multi-frame buffering of the second interface image can be performed through the interface image cache pointer according to the generation duration of each frame and the interface image caching rule.

[0106] As an example, the component of the composite image corresponding to the virtual display module is further used to calculate the caching duration for caching the virtual interface image based on the interface image caching rule and the generation duration of each frame of the second interface image. When the waiting duration meets the caching duration for caching the virtual interface image, the initial virtual interface image is obtained from the virtual interface image buffer through the virtual interface image cache pointer, and the virtual interface image cache pointer information corresponding to the initial virtual interface image is obtained. Then, it is determined whether the initial virtual interface image is complete according to the virtual interface image cache pointer information. When the initial virtual interface image is complete, the initial virtual interface image is used as the second interface image;

[0107] In a specific implementation, the component of the composite image corresponding to the virtual display module can obtain the second interface image from the virtual interface image buffer through the virtual interface image cache pointer, which is convenient for image synthesis. However, in order to ensure the integrity of the obtained second interface image, it is necessary to calculate the cache duration of the cached virtual interface image. When the time point arrives, data is obtained from the virtual interface image buffer, and complete cached data is retrieved. Therefore, the cache duration of the cached virtual interface image can be calculated based on the interface image cache rule and the generation duration of each frame. For example, if the interface image cache rule is to cache 4 frames of data and the generation duration of each frame is 20 ms, the calculated cache duration of the cached virtual interface image is 80 ms, and the maximum waiting time is 80 ms. For example, when the waiting duration reaches 80 ms, the initial virtual interface image can be obtained from the virtual interface image buffer through the virtual interface image cache pointer. The cache duration can also be 40 ms, 60 ms, etc. For example, if the cache duration is 40 ms, when the waiting duration is 40 ms, the initial virtual interface image can be obtained from the virtual interface image buffer through the virtual interface image cache pointer. However, the obtained initial virtual interface image is not necessarily a complete interface image. Therefore, it is also necessary to obtain the virtual interface image cache pointer information corresponding to the initial virtual interface image, that is, the position of the virtual interface image cache pointer or the feedback information sent by the virtual interface image cache pointer. For example, if the virtual interface image cache pointer is not at the position when caching is completed or the virtual interface image cache pointer does not send feedback information, it proves that the current frame of data has not been cached completely. Then, the previous frame of data can be obtained. If the current frame of data is cached completely, the current frame of data is used as the second interface image.

[0108] As an example, the component of the composite image corresponding to the virtual display module is further configured to calculate the cache duration of the cached camera image data based on the camera image cache rule and the generation duration of each frame of the camera image data. When the waiting duration meets the cache duration of the cached camera image data, the initial camera image data is obtained from the camera image data buffer through the camera image cache pointer, and the camera image cache pointer information corresponding to the initial camera image data is obtained. Then, it is determined whether the initial camera image data is complete based on the camera image cache pointer information. When the initial camera image data is complete, the initial camera image data is used as the camera image data.

[0109] The component of the composite image corresponding to the virtual display module is further configured to perform an overlay process on the camera image data and the second interface image to obtain the image data to be output.

[0110] In a specific implementation, after the component of the synthetic image corresponding to the virtual display module obtains the second interface image, it is also necessary to obtain the complete camera image data from the camera image data buffer to synthesize the second interface image and the camera image data. To ensure the integrity of the obtained camera image data, it is necessary to calculate the buffer duration of the buffered camera image data. When the time point arrives, the data is obtained from the camera image data buffer, and the complete buffered data is retrieved. Therefore, the component of the synthetic image corresponding to the virtual display module can calculate the buffer duration of the buffered camera image data through the camera image data buffer rule and the generation duration per frame. When the waiting duration reaches the buffer duration, the initial camera image data is obtained from the camera image data buffer through the camera image buffer pointer, and it is determined whether the initial camera image data is complete. If the initial image data is incomplete, the previous frame data of the initial image data can be obtained through the camera image buffer pointer and determined whether it is complete until the complete frame image data is obtained as the camera image data. If the initial camera image data is complete, the initial camera image data is used as the camera image data.

[0111] When the component of the synthetic image corresponding to the virtual display module obtains the complete camera image data and the second interface image through the camera image buffer pointer and the virtual interface image buffer pointer respectively, the hardware module can be called to implement image overlay processing to obtain the image data to be output. Through the overlay of images, various functions can be realized, including menus, icon prompts, system clocks, etc. When transmitting the camera image data and the second interface image, the pointer method is used for transmission, which improves the efficiency of image transmission and ensures low latency of image transmission.

[0112] As an example, the hardware abstraction layer of the Android system further includes: an encoder;

[0113] The encoder is used to receive the image data to be output transmitted by the synthetic component corresponding to the virtual display module, parse the image data to be output, determine the data format of the image data to be output, and when the data format of the image data to be output is the preset encoding format, perform encoding processing on the image data to be output in the preset encoding format to obtain the encoded image data to be output, and transmit the encoded image data to be output to the corresponding display terminal for display through the peripheral interface.

[0114] The encoder is used for image encoding. In a camera with a USB interface type, if the image data to be output is in encoding formats such as MJPGE, h.264, H.265, etc., the encoder needs to perform encoding processing. If it is in formats such as YUV, etc., the encoder directly skips. The image data to be output is sent to the peripheral interface through the relevant UVC driver. Then the encoder can receive the image data to be output transmitted by the synthesis component corresponding to the virtual reality module, and parse the image data to be output to determine the data format of the image data to be output. The preset encoding formats are encoding formats such as MJPGE, h.264, H.265, etc. When the data format of the image data to be output is the preset encoding format, it is necessary to encode the image data to be output. Therefore, the encoder can encode the image data to be output in the preset encoding format to obtain the encoded image data to be output, and transmit the encoded image data to be output to the corresponding display terminal through the peripheral interface for display.

[0115] This application provides a camera and an image processing method based on the Android system. Compared with the current image processing through the linux system, where the process of displaying the processed image interface is relatively cumbersome and complex and the cost is high, in this application, the image sensor collects the image data to be processed; and transmits the image data to be processed to the data processing module through the kernel layer of the Android system, so as to obtain the camera image data through preprocessing by the data processing module; the component for synthesizing the image corresponding to the display module receives the first interface image obtained by the display module and the camera image data, and performs superimposing processing on the camera image data and the first interface image to obtain the image data to be output. The image data to be output can be transmitted to the corresponding display terminal through the video interface for display. The component for synthesizing the image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superimposing processing on the camera image data and the second interface image to obtain the image data to be output, and transmits the image data to be output to the corresponding display terminal through the peripheral interface for display. It is possible to quickly obtain the interface image based on the Android system, facilitate the design, development and implementation of the interface image, provide a better human-computer interaction interface, and can capture the interface image generated by the surface through the virtual display module, and perform superimposing on the interface image and the camera image data through the component for synthesizing the image to obtain the image data to be output of the composite interface, as the video source of the camera, enhance the portability of the system, and display the image processing interface in real time during the superimposing process, with a simple operation process and reduced complexity of visual image processing.

[0116] Further, based on the first embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the component of the composite image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and the steps of performing superimposition processing on the camera image data and the second interface image to obtain the image data to be output include:

[0117] Step A1, obtain the refresh rate of the display terminal corresponding to the peripheral interface, the camera image caching rule, and the interface image caching rule;

[0118] In this embodiment, the interface image caching rule and the camera image caching rule can be set according to user requirements. To solve the system jitter problem, the camera image data buffer and the virtual interface image buffer can set multiple frames of buffering for the camera image data and the second interface image. The number of buffered frames can be set according to requirements, such as 4-frame buffering, 6-frame buffering, etc. This embodiment takes 4-frame buffering as an example for illustration. The refresh rate of the display terminal corresponding to the peripheral interface is the refresh rate of the display terminal of the USB interface type, such as a refresh rate of 50HZ, 100HZ, etc. This embodiment takes 50HZ as an example for illustration.

[0119] Step A2, calculate the generation duration of each frame of the camera image data and the generation duration of each frame of the second interface image through the refresh rate;

[0120] When the refresh rate is determined, the generation duration of each frame of the camera image data and the generation duration of each frame of the second interface image can be calculated according to the refresh rate. For example, if the refresh rate is 50HZ, the generation duration of each frame of the camera image data is 20ms, then a frame of camera image data is generated every 20ms, and the generation duration of each frame of the second interface image is 20ms, then a frame of the second interface image is generated every 20ms.

[0121] Step A3, calculate the caching duration of the cached camera image data through the generation duration of each frame of the camera image data and the camera image caching rule;

[0122] Step A4, calculate the caching duration of the cached virtual interface image through the generation duration of each frame of the second interface image and the interface image caching rule;

[0123] In this embodiment, to ensure the integrity of the obtained camera image data, it is necessary to calculate the caching duration of the cached camera image data and the caching duration of the cached virtual interface image. When the time point arrives, data is respectively obtained from the camera image data buffer and the virtual interface image buffer, and complete cached data is taken.

[0124] The caching duration of the cached camera image data can be calculated based on the camera image data caching rule and the generation duration per frame, and the caching duration of the cached virtual interface image can be calculated based on the interface image caching rule and the generation duration per frame. For example, if the interface image caching rule is to cache 4 frames of data and the generation duration per frame is 20 ms, then the calculated caching duration of the cached virtual interface image is 80 ms, and the longest waiting time is 80 ms. For example, when the waiting duration reaches 80 ms, the initial virtual interface image can be obtained from the virtual interface image buffer through the virtual interface image caching pointer. The caching duration can also be 40 ms, 60 ms, etc.

[0125] Step A5, when the waiting duration meets the caching duration of the cached virtual interface image, obtain the initial virtual interface image through the virtual interface image caching pointer, and when the waiting duration meets the caching duration of the cached camera image data, obtain the initial camera image data through the camera image caching pointer;

[0126] For example, if the caching duration is 40 ms, when the waiting duration is 40 ms, the initial virtual interface image can be obtained from the virtual interface image buffer through the virtual interface image caching pointer, and the initial camera image data can be obtained from the camera image buffer through the camera image caching pointer.

[0127] Step A6, obtain the virtual interface image caching pointer information corresponding to the initial virtual interface image and the camera image caching pointer information corresponding to the initial camera image data;

[0128] In a specific implementation, the obtained initial virtual interface image and initial camera image data are not necessarily complete interface image and camera image data. Therefore, it is also necessary to obtain the virtual interface image caching pointer information corresponding to the initial virtual interface image and the camera image caching pointer information corresponding to the initial camera image data. The virtual interface image caching pointer information can be the position where the virtual interface image caching pointer stays in the initial virtual interface image or the feedback information sent by the virtual interface image caching pointer, etc. The camera image caching pointer information can be the position where the camera image caching pointer stays in the initial camera image data or the feedback information sent by the camera image caching pointer, etc.

[0129] Step A7, determine whether the initial virtual interface image is complete according to the virtual interface image caching pointer information, and determine whether the initial camera image data is complete according to the camera image caching pointer information;

[0130] In this embodiment, the integrity of the acquired initial virtual interface image and initial camera image data can be determined by the virtual interface image cache pointer information and the camera image cache pointer information respectively. For example, if the position where the virtual interface image cache pointer stays is not at the position when caching is completed or the virtual interface image cache pointer does not send feedback information, it proves that the current frame data of the initial virtual interface image has not been cached completely. Then, the previous frame data can be acquired until the initial virtual interface image with complete caching is obtained. If the position where the virtual interface image cache pointer stays is not at the position of complete caching or the feedback information sent by the virtual interface image cache pointer is received, it means that the current frame data of the acquired initial virtual interface image is cached completely, and the current frame data is used as the second interface image. Similarly, the integrity of the initial camera data can be determined in the above manner.

[0131] Step A8, when the initial virtual interface image is complete, use the initial virtual interface image as the second interface image;

[0132] Step A9, when the initial camera image data is complete, use the initial camera image data as the camera image data;

[0133] In a specific implementation, when the initial virtual interface image is complete, the initial virtual interface image can be used as the second interface image. If the initial virtual interface image is incomplete, the previous frame of the virtual interface image can be re-acquired through the virtual interface image cache pointer, and it is determined whether the acquired previous frame of the virtual interface image is complete until a complete virtual interface image is obtained and used as the second interface image.

[0134] When the initial camera image data is complete, the initial camera image data can be used as the camera image data. If the initial camera image data is incomplete, the previous frame of the camera image can be re-acquired through the camera image cache pointer, and it is determined whether the acquired previous frame of the camera image is complete until a complete camera image is obtained and used as the camera image data.

[0135] Step A10, perform superposition processing on the second interface image and the camera image data to obtain the image data to be output.

[0136] In this embodiment, after the components of the composite image corresponding to the virtual display module respectively acquire the second interface image and the camera image data from the virtual interface image buffer area through the virtual interface image cache pointer and from the camera image buffer area through the camera image cache pointer, the hardware module can be called to implement image superposition processing to obtain the image data to be output.

[0137] As Figure 4 shown, Figure 4This is a schematic diagram of a scene involving a composite image. The camera image data cache and the virtual interface image cache each cache four frames of data. When the composite image component corresponding to the virtual display module needs to composite an image, the camera image data cache pointer retrieves data from the camera image data cache. The current caching progress is to cache the fourth frame of data, but the fourth frame of data has not yet been cached. Therefore, the camera image data cache pointer retrieves the third frame of data from the camera image data cache as the camera image data. The virtual interface image data cache pointer retrieves data from the virtual interface image cache. The current caching progress is to cache the third frame of data, but the third frame of data has not yet been cached. Therefore, the virtual interface image cache pointer retrieves the second frame of data from the virtual interface image cache as the second interface image, thereby outputting the camera image data and the second interface image to the composite image component for image overlay.

[0138] The component of the synthetic image corresponding to the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and superimposes the camera image data and the second interface image to obtain the image data to be output, and the method includes:

[0139] Step B1, generating a virtual interface image buffer creation instruction;

[0140] In this embodiment, since the USB type display terminal does not have a corresponding display interface, it is necessary to establish a virtual display interface of the virtual display module. Considering that the second interface image transmitted by the virtual display interface will be needed for image overlay in the future, the second interface image needs to be cached by managing the memory applied for the virtual display, so a virtual interface image cache area creation instruction is generated.

[0141] Step B2, allocating cache space in a preset memory address space according to the virtual interface image cache creation instruction to establish a virtual interface image cache;

[0142] As an example, after establishing a cache area, the component that synthesizes the image can obtain data in real time. When it is necessary to establish a virtual interface image cache area, cache space for caching the second interface image can be allocated in advance in the physical memory address space. The preset memory address space refers to the physical memory address space of the physical link layer of the Android system. Virtual memory address space can also be used, and this embodiment does not limit this.

[0143] A virtual interface image cache area is established by allocating cache space to cache the second interface image.

[0144] Specifically, for the convenience of obtaining data for the components of the synthetic image, a camera image data buffer also needs to be established for the camera image data to cache the camera image data. A camera image data buffer creation instruction can also be generated, and the camera image data buffer is created at the physical link layer of the Android system through the camera image data buffer creation instruction.

[0145] Step B3: Cache the second interface image obtained by the virtual display module into the virtual interface image buffer area through the virtual interface image buffer pointer.

[0146] In this embodiment, after the virtual interface image buffer area is established, the virtual interface image buffer pointer can be used to cache the second interface image obtained from the Virtual display interface of the virtual display module into the virtual interface image buffer area to cache the second interface image.

[0147] Step B4: Update the cache space after the caching is completed.

[0148] As an example, establishing the virtual interface image buffer area and the camera image data buffer can solve the problem of Android system jitter. When caching the second interface image, multiple frames of caching usually need to be considered, such as 4 frames, 6 frames, etc. The second interface image is written through the virtual interface image buffer pointer and the camera image data is written through the camera image data buffer pointer. After the writing is completed, the cache completion information is obtained, so as to update the cache space of the virtual interface image buffer area.

[0149] In this embodiment, the refresh rate of the display terminal corresponding to the peripheral interface, the camera image caching rule, and the interface image caching rule are obtained; the generation duration of each frame of the camera image data and the generation duration of each frame of the second interface image are calculated based on the refresh rate; the caching duration for caching the camera image data is calculated based on the generation duration of each frame of the camera image data and the camera image caching rule; the caching duration for caching the virtual interface image is calculated based on the generation duration of each frame of the second interface image and the interface image caching rule; when the waiting duration meets the caching duration for caching the virtual interface image, the initial virtual interface image is obtained through the virtual interface image caching pointer, and when the waiting duration meets the caching duration for caching the camera image data, the initial camera image data is obtained through the camera image caching pointer; the virtual interface image caching pointer information corresponding to the initial virtual interface image and the camera image caching pointer information corresponding to the initial camera image data are obtained; it is determined whether the initial virtual interface image is complete based on the virtual interface image caching pointer information, and it is determined whether the initial camera image data is complete based on the camera image caching pointer information; when the initial virtual interface image is complete, the initial virtual interface image is used as the second interface image; when the initial camera image data is complete, the initial camera image data is used as the camera image data; the second interface image and the camera image data are superimposed to obtain the image data to be output. By respectively obtaining the complete cached data in the buffer through the virtual interface image caching pointer and the camera image caching pointer, the problem that the image information used for synthesis by the components of the composite image is incomplete is avoided, the efficiency and effect of image synthesis are improved, and the data is transmitted in a pointer manner to improve the efficiency of image transmission.

[0150] In this embodiment, an instruction for creating a virtual interface image buffer is generated; a buffer space is allocated in the preset memory address space according to the instruction for creating the virtual interface image buffer to establish a virtual interface image buffer; the second interface image obtained by the virtual display module is cached in the virtual interface image buffer through the virtual interface image caching pointer; after the caching is completed, the buffer space is updated. By establishing the virtual interface image buffer, multiple frames of the second interface image can be cached, enabling the components of the composite image to quickly obtain the second interface image and improving the efficiency of the composite image components.

[0151] Further, based on the first embodiment and the second embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the steps before transmitting the image data to be output to the corresponding display terminal for display through the peripheral interface include:

[0152] Step D1, obtaining the UVC packet between the display terminal corresponding to the peripheral interface;

[0153] As an example, a UVC (USB Video Class) message refers to the message data between a UVC control unit and a display terminal of the USB interface type, and can interact with the display terminal corresponding to the peripheral interface through the UVC control unit to obtain the UVC message.

[0154] Step D2: Parse the UVC message to determine the transport protocol;

[0155] In this embodiment, the hardware abstraction layer of the Android system further includes a UVC control unit, which can parse the UVC message through the UVC control unit to obtain the corresponding transport protocol, and the transport protocol includes corresponding control or adjustment instructions. For example, when the display terminal of the USB interface type displays image data to be output and it is necessary to adjust the image data displayed on the display terminal, the display terminal sends a UVC message to the UVC control unit, and the UVC control unit parses the UVC message sent by the display terminal to obtain the transport protocol for adjusting the parameters.

[0156] Step D3: Generate a parameter adjustment instruction through the transport protocol;

[0157] As an example, the Android system further includes a control module, which can be located in the application layer of the Android system or in the hardware abstraction layer of the Android system, and the control module is connected to the UVC control unit. After obtaining the transport protocol, a parameter adjustment instruction can be generated through the transport protocol, and the UVC control unit sends the parameter adjustment instruction to the control module.

[0158] Step D4: Adjust the parameters during image sensor acquisition through the parameter adjustment instruction.

[0159] In this embodiment, the control module can adjust the parameters during image sensor acquisition according to the parameter adjustment instruction.

[0160] Further, when it is necessary to adjust the layout displayed by the camera, the display terminal can also interact with the UVC control unit to generate a corresponding layout adjustment instruction to the control module, and the control module adjusts the layout displayed by the camera according to the layout adjustment instruction.

[0161] Specifically, the hardware abstraction layer based on the Android system includes: an encoder; the encoder is used to receive the to-be-output image data transmitted by the composition component corresponding to the virtual display module, parse the to-be-output image data, determine the data format of the to-be-output image data, and when the data format of the to-be-output image data is a preset encoding format, perform encoding processing on the to-be-output image data in the preset encoding format to obtain the encoded to-be-output image data, and transmit the encoded to-be-output image data to the corresponding display terminal through the peripheral interface for display. The encoder is connected to the control module and can receive the encoding control instruction of the control module, so as to parse and encode the format of the to-be-output image data through the encoding control instruction.

[0162] The preset encoding formats are MJPGE encoding format, h.264 encoding format, and H.265 encoding format, and may also include other types of encoding formats. This embodiment does not limit this. When the data format of the to-be-output image data is a preset encoding format, it is necessary to perform encoding processing on this to-be-output image data. Therefore, the to-be-output image data can be parsed to determine the data format of the to-be-output image data.

[0163] In this embodiment, the data format of the to-be-output image, in addition to the preset encoding format, can also be the YUV format. When the data format of the to-be-output image data is a preset encoding format, the control module controls the encoder to perform encoding processing on the to-be-output image data in the preset encoding format according to the encoding control instruction to obtain the encoded to-be-output image data. The encoder transmits the encoded to-be-output image data to the corresponding display terminal through the peripheral interface for display.

[0164] As an example, after performing encoding processing on the to-be-output image data, the encoded to-be-output image data is transmitted to a display terminal of the USB interface type through the USB interface for display.

[0165] As an example, when the data format of the to-be-output image data is the YUV format, the to-be-output image data in the YUV format is transmitted to the corresponding display terminal through the peripheral interface for display. When the data format of the to-be-output image data is the YUV format, the encoder can be directly skipped, and the to-be-output image data in the YUV format is transmitted to the first display terminal through the USB interface for display.

[0166] In this embodiment, encoding processing can be performed according to the data format of the to-be-output image data, and the to-be-output image data can be transmitted to a display terminal of the USB interface type for display more quickly and securely.

[0167] Refer to Figure 5 , Figure 5It is a schematic diagram of the device structure of the hardware operating environment involved in the solution of the embodiment of this application.

[0168] As shown in Figure 5 , the image processing device based on the Android system may include: a processor 1001, a communication bus 1002, and a memory 1003. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1003.

[0169] Optionally, the image processing device based on the Android system may further include an image processing device based on the Android system.

[0170] Those skilled in the art can understand that Figure 5 the structure of the image processing device based on the Android system shown in

[0171] does not constitute a limitation on the image processing device based on the Android system, and may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Figure 5 As shown in

[0172] In the Figure 5 image processing device based on the Android system shown, the processor 1001 is used to execute the image processing program stored in the memory 1003 to implement the steps of the image processing method based on the Android system described in any one of the above.

[0173] The specific implementation manner of the image processing device based on the Android system in this application is basically the same as that of each embodiment of the above image processing method based on the Android system, and will not be repeated here.

[0174] The embodiment of this application provides a storage medium, and the storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to be used to implement the steps of the image processing method based on the Android system described in any one of the above.

[0175] The specific implementation of the storage medium of this application is basically the same as that of the above embodiments of the image processing method based on the Android system, and will not be elaborated here.

[0176] An embodiment of this application provides a camera. The camera runs the Android system. The physical link layer of the Android system includes an image sensor, a video interface, and a peripheral interface. The hardware abstraction layer of the Android system includes a data processing module and multiple components for synthesizing images. The application layer of the Android system includes at least two surfaces for displaying interface images. One surface obtains a first interface image through a display module, and the other surface obtains a second interface image through a virtual display module. The display module and the virtual display module respectively correspond one-to-one with the components for synthesizing images. The display module transmits the first interface image to the corresponding component for synthesizing images, and the virtual display module transmits the second interface image to the corresponding component for synthesizing images.

[0177] The image sensor is used to collect image data to be processed and transmit the image data to be processed to the data processing module through the kernel layer of the Android system.

[0178] The data processing module is used to preprocess the image data to be processed to obtain camera image data, and transmit the camera image data to the components for synthesizing images corresponding to the display module and the virtual display module respectively.

[0179] The component for synthesizing images corresponding to the display module is used to receive the first interface image obtained by the display module and the camera image data, and perform superposition processing on the camera image data and the first interface image to obtain image data to be output, and transmit the image data to be output to the corresponding display terminal through the video interface for display.

[0180] The component for synthesizing images corresponding to the virtual display module is used to receive the second interface image obtained by the virtual display module and the camera image data, and perform superposition processing on the camera image data and the second interface image to obtain image data to be output, and transmit the image data to be output to the corresponding display terminal through the peripheral interface for display.

[0181] In a possible implementation manner of this application, the physical link layer of the Android system further includes a virtual interface image buffer.

[0182] The virtual interface image buffer is used to obtain a buffer space allocated from a preset memory address space, and cache the second interface image obtained by the virtual display module through a virtual interface image buffer pointer to obtain a cached interface image, and update the buffer space after the caching is completed.

[0183] In a possible implementation manner of the present application, the physical link layer of the Android system further includes a camera image data buffer;

[0184] The camera image data buffer is used to obtain a camera image caching rule and the refresh rate of a display terminal corresponding to the peripheral interface, calculate the generation duration of each frame of the camera image data through the refresh rate, and perform multi-frame caching on the camera image data through a camera image buffer pointer based on the generation duration of each frame of the camera image data and the camera image caching rule;

[0185] The virtual interface image buffer is further used to obtain an interface image caching rule and the refresh rate of a display terminal corresponding to the peripheral interface, calculate the generation duration of each frame of the second interface image through the refresh rate, and perform multi-frame caching on the second interface image through a virtual interface image buffer pointer based on the generation duration of each frame of the second interface image and the interface image caching rule.

[0186] In a possible implementation manner of the present application, the component corresponding to the composite image of the virtual display module is further used to calculate the caching duration of the cached virtual interface image based on the interface image caching rule and the generation duration of each frame of the second interface image. When the waiting duration meets the caching duration of the cached virtual interface image, obtain an initial virtual interface image from the virtual interface image buffer through the virtual interface image buffer pointer, obtain the virtual interface image buffer pointer information corresponding to the initial virtual interface image, and determine whether the initial virtual interface image is complete according to the virtual interface image buffer pointer information. When the initial virtual interface image is complete, use the initial virtual interface image as the second interface image.

[0187] The component corresponding to the composite image of the virtual display module is further used to calculate the caching duration of the cached camera image data based on the camera image caching rule and the generation duration of each frame of the camera image data. When the waiting duration meets the caching duration of the cached camera image data, obtain initial camera image data from the camera image data buffer through the camera image buffer pointer, obtain the camera image buffer pointer information corresponding to the initial camera image data, and determine whether the initial camera image data is complete according to the camera image buffer pointer information. When the initial camera image data is complete, use the initial camera image data as the camera image data;

[0188] The component of the composite image corresponding to the virtual display module is further configured to superimpose the camera image data and the second interface image to obtain image data to be output.

[0189] In a possible implementation manner of the present application, the hardware abstraction layer of the Android system further includes: an encoder;

[0190] The encoder is configured to receive the image data to be output transmitted by the composite component corresponding to the virtual display module, parse the image data to be output to determine the data format of the image data to be output, and when the data format of the image data to be output is a preset encoding format, perform encoding processing on the image data to be output in the preset encoding format to obtain encoded image data to be output, and transmit the encoded image data to be output to a corresponding display terminal for display through the peripheral interface.

[0191] The specific implementation manners of the system of the present application are basically the same as those of the above embodiments of the image processing method based on the Android system, and will not be described in detail here.

[0192] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or system including that element.

[0193] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0194] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0195] The above are only the preferred embodiments of the present application, which do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

Claims

1. A camera, characterized in that, The camera runs the Android system. The physical link layer of the Android system includes an image sensor, a video interface, and a peripheral interface. The hardware abstraction layer of the Android system includes a data processing module and multiple components for synthesizing images. The application layer of the Android system includes two selectable surfaces for displaying interface images. One surface obtains a first interface image through a display module, and the other surface obtains a second interface image through a virtual display module. The display module and the virtual display module respectively correspond one-to-one with the components for synthesizing images. The display module transmits the first interface image to the corresponding component for synthesizing images, and the virtual display module transmits the second interface image to the corresponding component for synthesizing images; The image sensor is used to collect image data to be processed and transmit the image data to be processed to the data processing module through the kernel layer of the Android system; The data processing module is used to preprocess the image data to be processed to obtain camera image data, and transmit the camera image data to the corresponding components for synthesizing images of the display module and the virtual display module respectively; The component for synthesizing images corresponding to the display module is used to receive the first interface image obtained by the display module and the camera image data, and perform superimposition processing on the camera image data and the first interface image to obtain image data to be output, and transmit the image data to be output to the corresponding display terminal through the video interface for display; The component for synthesizing images corresponding to the virtual display module is used to receive the second interface image obtained by the virtual display module and the camera image data, and perform superimposition processing on the camera image data and the second interface image to obtain image data to be output, and transmit the image data to be output to the corresponding display terminal through the peripheral interface for display.

2. The camera according to claim 1, characterized in that, The physical link layer of the Android system further includes a virtual interface image buffer; The virtual interface image buffer is used to obtain a buffer space allocated by a preset memory address space, and cache the second interface image obtained by the virtual display module through a virtual interface image buffer pointer to obtain a cached interface image, and update the buffer space after caching is completed.

3. The camera according to claim 2, wherein The physical link layer of the Android system further includes a camera image data buffer; The camera image data buffer is used to obtain a camera image caching rule and the refresh rate of the display terminal corresponding to the peripheral interface, calculate the generation duration of each frame of the camera image data through the refresh rate, and perform multi-frame caching on the camera image data through a camera image buffer pointer based on the generation duration of each frame of the camera image data and the camera image caching rule; The virtual interface image buffer is further configured to obtain an interface image caching rule and the refresh rate of a display terminal corresponding to the peripheral interface, calculate the generation duration of each frame of the second interface image based on the refresh rate, and perform multi-frame caching on the second interface image through a virtual interface image cache pointer based on the generation duration of each frame of the second interface image and the interface image caching rule.

4. The camera according to claim 3, characterized in that, The component corresponding to the composite image of the virtual display module is further configured to calculate the caching duration of the cached virtual interface image based on the interface image caching rule and the generation duration of each frame of the second interface image, and when the waiting duration meets the caching duration of the cached virtual interface image, obtain an initial virtual interface image from the virtual interface image buffer through the virtual interface image cache pointer, obtain the virtual interface image cache pointer information corresponding to the initial virtual interface image, and determine whether the initial virtual interface image is complete according to the virtual interface image cache pointer information. When the initial virtual interface image is complete, use the initial virtual interface image as the second interface image. The component corresponding to the composite image of the virtual display module is further configured to calculate the caching duration of the cached camera image data based on the camera image caching rule and the generation duration of each frame of the camera image data, and when the waiting duration meets the caching duration of the cached camera image data, obtain initial camera image data from the camera image data buffer through the camera image cache pointer, obtain the camera image cache pointer information corresponding to the initial camera image data, and determine whether the initial camera image data is complete according to the camera image cache pointer information. When the initial camera image data is complete, use the initial camera image data as the camera image data. The component corresponding to the composite image of the virtual display module is further configured to perform an overlay process on the camera image data and the second interface image to obtain image data to be output.

5. The camera according to claim 1, characterized in that, The hardware abstraction layer of the Android system further includes: an encoder; The encoder is configured to receive the image data to be output transmitted by the component corresponding to the composite image of the virtual display module, parse the image data to be output to determine the data format of the image data to be output, and when the data format of the image data to be output is a preset encoding format, perform encoding processing on the image data to be output in the preset encoding format to obtain encoded image data to be output, and transmit the encoded image data to a corresponding display terminal through the peripheral interface for display.

6. An image processing method based on the Android system, characterized in that, The image processing method based on the Android system is applied to a camera according to any one of claims 1 to 5, and the image processing method based on the Android system includes: An image sensor collects image data to be processed and transmits the image data to be processed to a data processing module through the kernel layer of the Android system; The data processing module preprocesses the to-be-processed image data to obtain camera image data, and transmits the camera image data to components corresponding to the composite images of the display module and the virtual display module respectively; The component corresponding to the composite image of the display module receives the first interface image obtained by the display module and the camera image data, and performs superposition processing on the camera image data and the first interface image to obtain to-be-output image data; Transmit the to-be-output image data to the corresponding display terminal for display through a video interface; The component corresponding to the composite image of the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superposition processing on the camera image data and the second interface image to obtain to-be-output image data; Transmit the to-be-output image data to the corresponding display terminal for display through a peripheral interface; 7. The image processing method based on the Android system according to claim 6, wherein Before the component corresponding to the composite image of the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superposition processing on the camera image data and the second interface image to obtain to-be-output image data, it further includes: Generate a virtual interface image buffer creation instruction; Allocate a buffer space in a preset memory address space according to the virtual interface image buffer creation instruction to establish a virtual interface image buffer; Cache the second interface image obtained by the virtual display module to the virtual interface image buffer through a virtual interface image buffer pointer; After caching is completed, update the buffer space.

8. The image processing method based on the Android system according to claim 6, wherein, The component corresponding to the composite image of the virtual display module receives the second interface image obtained by the virtual display module and the camera image data, and performs superposition processing on the camera image data and the second interface image to obtain to-be-output image data, including: Obtain the refresh rate of the display terminal corresponding to the peripheral interface, the camera image caching rule, and the interface image caching rule; Calculate the generation duration of each frame of the camera image data and the generation duration of each frame of the second interface image through the refresh rate; Calculate the caching duration for caching the camera image data through the generation duration of each frame of the camera image data and the camera image caching rule; Calculate the caching duration for caching the virtual interface image through the generation duration of each frame of the second interface image and the interface image caching rule; When the waiting duration meets the caching duration for caching the virtual interface image, obtain the initial virtual interface image through the virtual interface image buffer pointer, and when the waiting duration meets the caching duration for caching the camera image data, obtain the initial camera image data through the camera image buffer pointer; Obtain the virtual interface image buffer pointer information corresponding to the initial virtual interface image and the camera image buffer pointer information corresponding to the initial camera image data; Judge whether the initial virtual interface image is complete according to the virtual interface image buffer pointer information, and judge whether the initial camera image data is complete according to the camera image buffer pointer information; When the initial virtual interface image is complete, use the initial virtual interface image as the second interface image; When the initial camera image data is complete, use the initial camera image data as the camera image data; Perform superposition processing on the second interface image and the camera image data to obtain the image data to be output.

9. The image processing method based on the Android system according to claim 6, characterized in that, Before transmitting the image data to be output to the corresponding display terminal for display through the peripheral interface, it further includes: Obtain the UVC message between the peripheral interface and the corresponding display terminal; Parse the UVC message to determine the transmission protocol; Generate a parameter adjustment instruction through the transmission protocol; Adjust the parameters during image sensor acquisition through the parameter adjustment instruction.

10. The image processing method based on the Android system according to claim 9, characterized in that, Transmitting the image data to be output to the corresponding display terminal for display through the peripheral interface includes: Parse the image data to be output to determine the data format of the image data to be output; When the data format of the image data to be output is the YUV format, transmit the image data to be output in the YUV format to the corresponding display terminal for display through the peripheral interface; When the data format of the image data to be output is a preset encoding format, generate an encoding control instruction through the transmission protocol; Perform encoding processing on the image data to be output in the preset encoding format through the encoding control instruction to obtain the encoded image data to be output; Transmit the encoded image data to be output to the corresponding display terminal for display through the peripheral interface.

Images