Display device and image processing method

Abstract

The embodiment of the invention discloses a display device and an image processing method, the display device only comprises a front camera, a target application receives an operation of starting a shooting function, and when it is determined that a virtual rear camera is to be used, a first shooting preview instruction is sent to a preview processing module, the first shooting preview instruction comprises a first camera identifier corresponding to a rear camera; the preview processing module responds to the first shooting preview instruction and sends a target camera starting instruction to the driving layer; the driving layer responds to the target camera starting instruction, starts a front-facing camera, and stores a preview stream collected by the front-facing camera to a first video node; and the preview processing module acquires a first rotation angle based on the first camera identifier and the current screen direction, and performs rotation processing on a preview stream acquired from the first video node according to the first rotation angle. Therefore, for the display equipment only provided with the front camera, the application program can be supported to carry out rear shooting, and the preview image can be ensured to be accurately displayed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a display device and an image processing method. Background Technology

[0002] Display devices can install applications with shooting capabilities, such as camera apps, photo editing apps, and QR code scanning apps. These applications may enable the rear camera for shooting. However, some display devices, such as tablets, may only have a physical front-facing camera and no physical rear camera. In this case, the application cannot enable the rear camera, and the display device's shooting preview interface will show a black screen or indicate that rear shooting is not supported, rendering the application's shooting function unusable.

[0003] For display devices equipped with only a front-facing physical camera, it is necessary to consider how to support applications to take photos with the rear camera, and ensure that the preview and captured images can be displayed accurately without problems such as abnormal image rotation angles, such as inverted images. Summary of the Invention

[0004] Some embodiments of this application provide a display device and an image processing method. For display devices that are only configured with a front-facing physical camera, a virtual rear camera is generated by creating a virtual video node in the driver layer to meet the application's need to enable the rear camera for shooting. When the application enables the rear camera, a rotation angle is matched based on the camera category identifier and the current screen orientation of the display device, and the preview stream captured by the front camera is rotated based on the rotation angle to ensure that the image of the preview interface captured under the current screen orientation will not have problems such as inversion.

[0005] In a first aspect, some embodiments of this application provide a display device, including:

[0006] A front-facing camera, configured to capture images from one side of the screen;

[0007] The display is configured to show the shooting preview interface;

[0008] A controller, coupled to the display and the front-facing camera, runs a preview processing module, a driver layer, and a target application, the target application having a shooting function. The controller is configured to:

[0009] In response to receiving an operation to activate the shooting function of the target application, the target camera to be used by the target application is determined; the target camera is one of the front camera and the rear camera, and the rear camera is a virtual camera generated by the driver layer by creating a first video node, the first video node being a virtual video node associated with the rear camera;

[0010] When the target camera is determined to be the rear camera, a first shooting preview instruction is sent to the preview processing module through the target application. The first shooting preview instruction includes a first camera identifier corresponding to the rear camera.

[0011] In response to the first shooting preview command, the preview processing module sends a command to start the target camera to the driver layer.

[0012] In response to the command to start the target camera, the driver layer starts the front camera and stores the first preview stream captured by the front camera into the first video node.

[0013] The preview processing module obtains a first rotation angle based on the first camera identifier and the current screen orientation of the display device. The first rotation angle is a rotation angle matched by the preview processing module. The rotation angle is the angle at which the original image captured by the front camera is rotated clockwise to be consistent with the current screen orientation.

[0014] The preview processing module obtains a first preview stream from the first video node, rotates the first preview stream according to the first rotation angle to obtain a second preview stream, and transmits the second preview stream to the display so that the display shows the second preview stream on the shooting preview interface.

[0015] The beneficial effects of the above-mentioned first aspect of the embodiment are as follows: To meet the rear-facing shooting needs of some applications, the driver layer pre-creates a virtual video node to generate a virtual rear camera, so that the display device has a physical front camera and a virtual rear camera; when the application starts the shooting function, it determines which type of camera to use. If the target application wants to enable the rear camera, it sends a first shooting preview instruction to the preview processing module. The first shooting preview instruction includes a first camera identifier corresponding to the rear camera. In this way, the preview processing module knows that the rear camera is about to be enabled by reading the first camera identifier in the instruction. The rear camera is a virtual camera (without actual image capture capability), requiring the front camera to be activated. The preview stream captured by the front camera is stored in a virtual video node, thus simulating the shooting of the rear camera. In the native Android camera processing logic, the target application automatically sends a rotation angle when it determines to use the rear camera. After rotating the preview stream based on this rotation angle, the shooting preview image can be made to present an upright effect consistent with the screen orientation. However, in this application, the rear camera is virtual, and the front camera is actually responsible for shooting. This results in the shooting preview image being inverted after using the rotation angle sent by the target application.

[0016] Therefore, in this application, the preview processing module matches the correct rotation angle (i.e., the first rotation angle) based on the camera type and screen orientation, instead of using the rotation angle issued by the target application. Based on the first camera identifier, the preview processing module identifies that the target application is using a rear camera. Combined with the current screen orientation of the display device (landscape or portrait), it obtains the first rotation angle and obtains the first preview stream from the virtual video node. It then uses the first rotation angle to rotate the first preview stream, ultimately causing the display to show the processed second preview stream on the shooting preview interface. In this way, for display devices that are only configured with a front physical camera, by creating a virtual rear camera, the problem of black screen caused by missing hardware configuration for rear shooting is solved. This allows applications to enable rear shooting and avoids the abnormal rotation angle caused by the virtual rear camera not having image acquisition capabilities, where the front camera is still responsible for image acquisition when the rear camera is enabled. This prevents image inversion and ensures the accuracy of the shooting preview interface display.

[0017] In some embodiments of the first aspect, the display device further includes a gyroscope and a graphics processor; before sending a first shooting preview command to the preview processing module by the target application to determine the target camera to be used by the target application, the controller is further configured to: when determining that the target camera is the rear camera, obtain a first angle through the target application, the first angle being the angle by which the mounting angle of the gyroscope is rotated clockwise to match the imaging angle of the rear camera; obtain a screen rendering angle mapped by the current screen orientation of the display device through the target application, the screen rendering angle being the angle between the image rendering direction of the graphics processor and the screen orientation; calculate a second rotation angle through the target application based on the first angle and the screen rendering angle; and generate the first shooting preview command through the target application, the first shooting preview command further including the second rotation angle.

[0018] The beneficial effects of this embodiment are as follows: When the target application determines that the rear camera will be enabled, since the target application cannot perceive whether the rear camera is virtual or physical, it will follow the native Android camera logic to obtain the rotation angle when the gyroscope's installation angle is rotated clockwise to match the imaging angle of the rear camera, i.e., obtain the first angle; the target application will also determine the angle between the graphics processor's image rendering direction and the screen direction based on the current screen orientation, i.e., the screen rendering angle; based on the first angle and the screen rendering angle, the target application will calculate the second rotation angle according to the default Android formula for calculating the rear camera rotation angle. This second rotation angle is the rotation angle calculated by the target application and sent to the preview processing module through the first shooting preview command. In this way, the target application still calculates and sends the second rotation angle according to the original logic without changing the target application's software program configuration. Subsequently, the preview processing module will determine whether the rotation angle sent by the target application needs to be corrected by comparing the first rotation angle and the second rotation angle.

[0019] In some embodiments of the first aspect, the preview processing module obtains a first preview stream from the first video node, and rotates the first preview stream according to the first rotation angle to obtain a second preview stream. Specifically, the preview processing module compares the first rotation angle and the second rotation angle; when the second rotation angle and the first rotation angle are different, the preview processing module rotates the first preview stream according to the first rotation angle to obtain the second preview stream.

[0020] The beneficial effects of this embodiment are as follows: The preview processing module obtains a first rotation angle through a preset mapping rule and reads a second rotation angle from the first shooting preview command, and compares whether the first rotation angle and the second rotation angle are consistent. If the first rotation angle and the second rotation angle are consistent, it means that the rotation angle issued by the target application is correct and there is no need to correct the rotation angle. Therefore, the preview stream can be rotated based on the second rotation angle. If the first rotation angle and the second rotation angle are inconsistent, it means that the second rotation angle issued by the target application is incorrect. If the second rotation angle is used to rotate the preview stream, it will cause the image displayed on the shooting preview interface to be inverted. Therefore, it is necessary to calibrate the rotation angle, that is, to use the first rotation angle to rotate the preview stream, thereby avoiding image inversion and ensuring the accuracy of the shooting preview interface display.

[0021] In some embodiments of the first aspect, the first shooting preview instruction further includes the application identifier of the target application; the preview processing module obtains a first rotation angle based on the first camera identifier and the current screen orientation of the display device, specifically configured to: query a preset mapping table for a matching first rotation angle based on the application identifier of the target application, the first camera identifier, and the current screen orientation of the display device; wherein, the preset mapping table includes rotation angles mapped by multiple shooting scene information, and the shooting scene information includes the application identifier of an application with shooting function, the camera identifier, and the screen orientation.

[0022] The beneficial effects of this embodiment are as follows: The display device is first configured with a preset mapping table, which specifies the correct rotation angle to be used in different shooting scenarios. The shooting scenario involves the application category, camera category (front camera and rear camera), and screen orientation (landscape and portrait). Based on the application identifier of the currently running target application (the application identifier is used to identify the application category), the first camera identifier (used to represent the rear camera), and the current screen orientation, the first rotation angle can be accurately mapped and matched, so that the rotation angle conforms to the conditions of the current shooting scenario, ensuring the accuracy of the shooting preview and avoiding problems such as image inversion caused by abnormal rotation angle.

[0023] In some embodiments of the first aspect, the shooting scene information further includes the functional state of the display device, the functional state including at least one of the on / off state of the automatic rotation function and the on / off state of the forced landscape function; when the automatic rotation function is on, the image displayed on the shooting preview interface rotates synchronously with the screen; when the forced landscape function is on, the image displayed on the shooting preview interface is rotated to the landscape orientation; the preview processing module obtains a first rotation angle based on the rear camera and the current screen orientation of the display device, specifically configured to: query the preset mapping table for a matching first rotation angle based on the application identifier of the target application, the first camera identifier, the current screen orientation of the display device, and the current functional state of the display device.

[0024] The beneficial effects of this embodiment are as follows: Some applications with shooting functions may only support portrait shooting and not landscape shooting. To adapt to the needs of landscape shooting, the display device can be configured with at least one of automatic rotation function and forced landscape function. Users can set the function status of automatic rotation function and / or forced landscape function according to their usage needs. Among them, the automatic rotation function allows the preview image to automatically rotate with the screen orientation. For example, when the user switches the screen from portrait to landscape, the preview image also needs to be rotated to the landscape orientation according to the corresponding rotation angle. The forced landscape function forces the preview image to be rotated to the landscape orientation regardless of whether the screen orientation is portrait or landscape. Such automatic rotation or forced rotation methods may lead to problems such as abnormal preview orientation and preview ratio distortion. Therefore, when setting the preset mapping table, the functional states of automatic rotation and / or forced landscape mode can also be considered in the shooting scene, ultimately providing a more complete and accurate mapping relationship. Based on the application identifier of the target application, the category identifier of the rear camera, the current screen orientation, and the current functional state, the preview processing module can accurately map and match the first rotation angle, so that the rotation angle conforms to the conditions of the current shooting scene, avoiding image inversion caused by abnormal rotation angle, and avoiding abnormal preview orientation and preview ratio imbalance caused by forced landscape mode, ensuring the accuracy of shooting preview and improving the display effect of shooting preview interface.

[0025] In some embodiments of the first aspect, after determining the target camera to be used by the target application in response to receiving an operation to activate the shooting function of the target application, the controller is further configured to: when the target camera is determined to be the front-facing camera, obtain a second angle through the target application, the second angle being the angle rotated clockwise to match the imaging angle of the front-facing camera; calculate a third rotation angle through the target application based on the second angle and the screen rendering angle; and send a second shooting preview instruction to the preview processing module through the target application, the second shooting preview instruction including a second camera identifier corresponding to the front-facing camera and the third rotation angle.

[0026] The beneficial effects of this embodiment are as follows: When the target application determines that the front-facing camera will be enabled, it will, according to the native Android camera logic, obtain the rotation angle when the gyroscope's installation angle is rotated clockwise to match the imaging angle of the front-facing camera, i.e., obtain the second angle. Then, based on the second angle and the screen rendering angle, it calculates the third rotation angle according to the default Android formula for calculating the front-facing camera rotation angle. This third rotation angle is calculated by the target application and sent to the preview processing module through the second shooting preview command. Since the front-facing camera is a real physical camera, the third rotation angle is calculated based on the front-facing camera, and the preview stream is also captured by the front-facing camera, the rotation angle sent by the target application is not abnormal when the front-facing camera is enabled, and will not cause problems such as the image displayed in the shooting preview interface being inverted. In this way, the target application still calculates and sends the third rotation angle according to the original logic, without changing the target application's software program configuration. Subsequently, the preview processing module processes the preview stream based on the third rotation angle.

[0027] In some embodiments of the first aspect, after sending a second shooting preview instruction to the preview processing module via the target application, the controller is further configured to: send a target camera activation instruction to the driver layer in response to the second shooting preview instruction via the preview processing module; activate the front-facing camera in response to the target camera activation instruction via the driver layer, store the first preview stream captured by the front-facing camera to a second video node, the second video node being a real video node associated with the front-facing camera; obtain the first preview stream from the second video node via the preview processing module, rotate the first preview stream according to the third rotation angle to obtain a third preview stream, and transmit the third preview stream to the display so that the display shows the third preview stream on the shooting preview interface.

[0028] The beneficial effects of this embodiment are as follows: The preview processing module, responding to the second shooting preview command, can read the third rotation angle from the second shooting preview command and send a command to the driver layer to start the target camera; the driver layer, responding to the target camera command, starts the physical front-facing camera. Since the target application is using the front-facing camera, the preview stream captured by the front-facing camera will be stored in the second video node (the real video node); the preview processing module can obtain the preview stream from the second video node and perform rotation processing on the preview stream based on the third rotation angle. In this way, since there will be no abnormal rotation angle when the target application activates the front-facing camera, the preview processing module does not need to match the rotation angle separately according to a preset mapping table, but directly uses the third rotation angle sent by the target application to process the preview stream, thereby reducing the number of procedures executed by the preview processing module and improving image processing efficiency while ensuring accurate shooting preview.

[0029] In some embodiments of the first aspect, the controller further operates a photo processing module, and the controller is further configured to: during the display of the second preview stream on the photo preview interface, in response to receiving an image capture operation, send a photo capture command to the photo processing module through the target application, the photo capture command including a first camera identifier corresponding to the rear camera and a target timestamp, the target timestamp being used to characterize the time when the target application receives the image capture operation; in response to the photo capture command, the photo processing module obtains image frame data and thumbnail data corresponding to the target timestamp from the first video node; processes the image frame data through the photo processing module to generate an original captured image in a preset format, and rotates the original captured image based on the first rotation angle to obtain a target main image; processes the thumbnail data through the photo processing module to generate an original thumbnail corresponding to the original captured image, and rotates the original thumbnail based on the first rotation angle to obtain a target thumbnail.

[0030] The beneficial effects of this embodiment are as follows: Both previewing and taking photos require image rotation, and the same rotation angle is typically used. This means that when the target application uses the rear camera, not only will the image be inverted due to abnormal rotation angles during previewing, but the main image and thumbnail will also be inverted when the user clicks the shutter button to take a photo. To address this, this application obtains image frame data and thumbnail data corresponding to the target timestamp from a virtual video node during the photo-taking stage. The photo-taking processing module rotates the image frames and thumbnails based on a first rotation angle to obtain the target main image and target thumbnail. In this way, the captured main image and its corresponding thumbnail can be processed using the correct rotation angle, avoiding image inversion caused by incorrect rotation angles and improving the accuracy of the photo-taking angle and the image display effect.

[0031] In some embodiments of the first aspect, the controller further operates a hardware abstraction layer, and before determining the target camera to be used by the target application in response to receiving an operation to initiate the shooting function of the target application, the controller is further configured to: send a video node list to the hardware abstraction layer via the driver layer after creating the first video node, the video node list including first node configuration information and second node configuration information, the first node configuration information including a first node address and a first node ID mapped to the first video node, and the second node configuration information including a second node address and a second node ID mapped to the second video node; upon receiving the video node list via the hardware abstraction layer, set a rear camera attribute for the first video node and assign a first camera identifier to the first video node, the first camera identifier being used to indicate that the first video node is associated with the rear camera; set a front camera attribute for the second video node via the hardware abstraction layer and assign a second camera identifier to the second video node, the second camera identifier being used to indicate that the second video node is associated with the front camera; and store a camera list via the hardware abstraction layer, the camera list including: a mapping relationship between the first node configuration information and the first camera identifier, and a mapping relationship between the second node configuration information and the second camera identifier.

[0032] The beneficial effects of this embodiment are as follows: Since the front-facing camera is hardware with a real physical structure, the driver layer automatically creates a corresponding second video node for the front-facing camera. When a virtual camera needs to be generated, the driver layer creates a first and second video node and maintains a video node list, which records the relevant configuration information of the camera's video nodes. The hardware abstraction layer receives the video node list, sets the facing attribute of the first video node to back (rear-facing), and assigns a first camera identifier (e.g., "0" indicates a rear-facing camera) to the first video node. It also sets the facing attribute of the second video node to front (front-facing) and assigns a second camera identifier (e.g., "1" indicates a front-facing camera) to the second video node. Thus, the hardware abstraction layer can maintain a camera list, thereby recording the shooting capabilities of the display device. Before determining the target camera, the target application can access the hardware abstraction layer to read the camera list. When the camera list includes the first camera identifier, it assumes that the display device is configured with a rear-facing camera and then enables the rear-facing camera. In this way, a virtual video node is created through the driver layer, and then the hardware abstraction layer configures the virtual video node as a rear camera attribute, thereby generating a virtual rear camera. The front camera assists the virtual rear camera in image acquisition, so as to meet the application's needs for rear shooting and avoid the situation where the shooting preview interface is black and the application's shooting function cannot be used.

[0033] Secondly, some embodiments of this application also provide an image processing method in a display device, the display device being equipped with a front-facing camera, the method comprising:

[0034] The target application receives an operation to start the shooting function and determines the target camera to be used; the target camera is one of the front camera and the rear camera, and the rear camera is a virtual camera generated by the driver layer by creating a first video node, the first video node being a virtual video node associated with the rear camera;

[0035] When the target application determines that the target camera is a rear camera, it sends a first shooting preview instruction to the preview processing module. The first shooting preview instruction includes a first camera identifier corresponding to the rear camera.

[0036] In response to the first shooting preview command, the preview processing module sends a command to start the target camera to the driver layer;

[0037] In response to the command to start the target camera, the driver layer starts the front camera and stores the first preview stream captured by the front camera into the first video node;

[0038] The preview processing module obtains a first rotation angle based on the first camera identifier and the current screen orientation of the display device. The first rotation angle is a rotation angle matched by the preview processing module. The rotation angle is the angle at which the original image captured by the front camera is rotated clockwise to be consistent with the current screen orientation.

[0039] The preview processing module obtains a first preview stream from the first video node, rotates the first preview stream according to the first rotation angle to obtain a second preview stream, and transmits the second preview stream to the display so that the display shows the second preview stream on the shooting preview interface.

[0040] The beneficial effects of the second aspect of the embodiment described above are as follows: To meet the rear-facing camera requirements of some applications, the driver layer pre-creates a virtual video node to generate a virtual rear camera, thus giving the display device a physical front camera and a virtual rear camera. When an application starts the shooting function, it determines which type of camera to use. If the target application wants to enable the rear camera, it sends a first shooting preview command to the preview processing module. This first shooting preview command includes a first camera identifier corresponding to the rear camera. The preview processing module then reads the first camera identifier in the command to determine that the rear camera is about to be enabled. The rear camera is a virtual camera (without actual image capture capability), requiring the front camera to be activated. The preview stream captured by the front camera is stored in a virtual video node, thus simulating the shooting of the rear camera. In the native Android camera processing logic, the target application automatically sends a rotation angle when it determines to use the rear camera. After rotating the preview stream based on this rotation angle, the shooting preview image can be made to present an upright effect consistent with the screen orientation. However, in this application, the rear camera is virtual, and the front camera is actually responsible for shooting. This results in the shooting preview image being inverted after using the rotation angle sent by the target application.

[0041] Therefore, in this application, the preview processing module matches the correct rotation angle (i.e., the first rotation angle) based on the camera type and screen orientation, instead of using the rotation angle issued by the target application. Based on the first camera identifier, the preview processing module identifies that the target application is using a rear camera. Combined with the current screen orientation of the display device (landscape or portrait), it obtains the first rotation angle and obtains the first preview stream from the virtual video node. It then uses the first rotation angle to rotate the first preview stream, ultimately causing the display to show the processed second preview stream on the shooting preview interface. In this way, for display devices that are only configured with a front physical camera, by creating a virtual rear camera, the problem of black screen caused by missing hardware configuration for rear shooting is solved. This allows applications to enable rear shooting and avoids the abnormal rotation angle caused by the virtual rear camera not having image acquisition capabilities, where the front camera is still responsible for image acquisition when the rear camera is enabled. This prevents image inversion and ensures the accuracy of the shooting preview interface display. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in some embodiments of this application or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1A A schematic diagram of the structure of a display device provided in some embodiments of this application;

[0044] Figure 1B Schematic diagram of the structure of the display device provided in some embodiments of this application Figure Two ;

[0045] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0046] Figure 3 This application provides schematic diagrams illustrating the software configuration of a display device operating system in some embodiments.

[0047] Figure 4 A schematic diagram of a shooting preview interface provided for some embodiments of this application;

[0048] Figure 5 A schematic diagram illustrating the principle of generating a virtual camera for a display device 100 provided in some embodiments of this application;

[0049] Figure 6 Comparison images of the preview effects of the front camera and the rear camera provided in some embodiments of this application;

[0050] Figure 7 A schematic diagram showing the installation angle of a gyroscope provided in some embodiments of this application;

[0051] Figure 8 A schematic diagram illustrating the imaging angle of a camera provided in some embodiments of this application;

[0052] Figure 9 A schematic diagram illustrating the calculation principle of the first and second angles provided for some embodiments of this application;

[0053] Figure 10 A schematic diagram illustrating image inversion when a target application using a rear camera is used, as provided in some embodiments of this application;

[0054] Figure 11 Illustrations of image inversion occurring when the target application using the rear camera, provided in some embodiments of this application. Figure Two ;

[0055] Figure 12 A flowchart of an image processing method provided in some embodiments of this application;

[0056] Figure 13 A schematic diagram illustrating how image inversion is eliminated by correcting the rotation angle when a virtual rear camera is used in a target application provided in some embodiments of this application;

[0057] Figure 14 This illustration shows how image inversion is eliminated by correcting the rotation angle when using a virtual rear camera in target applications provided in some embodiments of this application. Figure Two ;

[0058] Figure 15 This is a schematic diagram of the preview interface captured after the forced landscape function is enabled when the display device 100 provided in some embodiments of this application is in portrait mode;

[0059] Figure 16 Flowchart of the image processing method provided in some embodiments of this application Figure Two ;

[0060] Figure 17 Flowchart of the image processing method provided in some embodiments of this application Figure Three ;

[0061] Figure 18 Flowchart of the image processing method provided in some embodiments of this application Figure Four ;

[0062] Figure 19 A schematic diagram of the user interface when taking a photo is completed, provided for some embodiments of this application;

[0063] Figure 20 A schematic diagram of the user interface when taking a photo is completed, provided for some embodiments of this application. Figure Two ;

[0064] Figure 21 Flowchart of the image processing method provided in some embodiments of this application Figure Five . Detailed Implementation

[0065] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0066] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0067] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0068] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0069] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0070] In this application embodiment, "display device" generally refers to a device with screen display, shooting, and data processing capabilities. For example, display devices include, but are not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, and augmented reality devices.

[0071] Figure 1A A schematic diagram of the structure of a display device provided in some embodiments of this application. See also... Figure 1AThe display device can be a large-screen device such as a smart TV. Taking a smart TV as an example, it can be equipped with a camera to capture images of the scene within its field of view. Smart TVs are generally mounted on a TV wall or fixed to a base bracket near the TV wall, such as... Figure 1A Smart TVs typically include a front-facing camera 120a, which is a piece of hardware with a real physical structure used to capture images from one side of the smart TV screen to enable functions such as facial recognition, gesture recognition, follow-up exercises, video calls, and taking photos.

[0072] Figure 1B Schematic diagram of the structure of the display device provided in some embodiments of this application Figure Two The display device can also be a smartphone, tablet, or other mobile terminal. A smartphone has a front-facing camera 120a located on the side closer to the screen (away from the back panel), and a rear-facing camera can be located on the side closer to the back panel (away from the screen). See also Figure 1B For devices such as tablets and electronic barcode scanners, there is usually only one physical front-facing camera (120a), and no physical rear-facing camera. These devices only support front-facing camera shooting and do not have rear-facing shooting capabilities.

[0073] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application.

[0074] In some embodiments, see Figure 2 The display device 100 may include, but is not limited to, at least one of the following: a communication device 110, a detector 120, a controller 130, a display 140, an audio output device 150, a memory, a power supply, and a user input interface.

[0075] In some embodiments, the communication device 110 is a component used to communicate with external devices or servers according to various communication protocol types. The display device 100 may have multiple communication devices 110 depending on the supported communication methods. For example, when the display device 100 supports wireless network communication, it may have a communication device 110 with WiFi functionality. When the display device 100 supports Bluetooth connection communication, it may have a communication device 110 with Bluetooth functionality.

[0076] In some embodiments, the communication device 110 enables the display device 100 to communicate with external devices or servers via wireless or wired connections. Wired connections utilize components such as data cables and interfaces to connect the display device 100 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 100 can directly establish a connection with external devices or indirectly establish a connection through gateways, routers, or connection devices.

[0077] In some embodiments, the detector 120 is used to collect signals from the external environment or signals interacting with the outside world. For example, the detector 120 may include a sound acquisition device, such as a microphone, for receiving external sounds. The detector 120 includes a front-facing camera 120a, which is used to capture images of the external environment scene on one side of the screen to achieve functions such as facial recognition, gesture recognition, follow-up exercises, video calls, video recording, and taking photos. The detector 120 may include a gyroscope 120b, which can measure the angular velocity of the display device 100 to detect the rotation state of the display device 100 in space. In this embodiment, the detector 120 does not include a rear-facing camera with a physical hardware structure, but only includes the front-facing camera 120a.

[0078] In some embodiments, the display 140 includes display function components for presenting an image and driving components for driving the image display. The display 140 is used to receive and display image signals output from the controller 130. For example, the display 140 can be used to display video content, image content, menu control interface components, and user interface (UI) interfaces. After the controller 130 starts and runs a target application with a shooting function, the display 140 can display the corresponding application page. When the target application is running, the controller 130, in response to the operation of starting the shooting function, can control the display 140 to display a shooting preview interface.

[0079] In some embodiments, the controller 130 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processing unit (GPU), and a power processor, and a first to an nth interface for input / output. The controller 130 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 130 controls the overall operation of the display device 100.

[0080] In some embodiments, the audio output device 150 can be the built-in speaker of the display device 100 or an external audio output device (such as a Bluetooth headset, Bluetooth speaker, etc.) connected to the display device 100. For external audio output devices connected to the display device 100, the display device 100 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 100 to output sound from the display device 100.

[0081] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 140, and the user input interface receives user input commands through the graphical user interface (GUI).

[0082] In some embodiments, the user input interface can be used to receive instructions from user input. If the display device 100 supports touch functionality, for example, if the display device 100 is a tablet device, then the user input interface can be a touch-sensitive interface.

[0083] In order to perform user interaction, in some embodiments, the display device 100 may run an operating system. An operating system is a computer program used to manage and control the hardware and software resources in the display device 100. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 100.

[0084] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0085] Figure 3 This is a schematic diagram illustrating the software configuration of the operating system of a display device provided in some embodiments of this application.

[0086] In some embodiments, the operating system can be divided into different modules or levels based on the functions it implements. For example... Figure 3 As shown, the operating system can be divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Runtime Library layer, and the Kernel layer.

[0087] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 100 to run the applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0088] In some embodiments, see Figure 3 The application layer can run multiple applications, such as application 1, application 2, and application 3. To achieve functions such as image capture, image processing, and image display, the application layer can run applications with shooting capabilities, such as camera applications and barcode scanning applications. The controller can implement the technical solutions provided in the embodiments of this application by running at least one related application in the application layer.

[0089] In some embodiments, see Figure 3 The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0090] In some embodiments, see Figure 3 The framework layer includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. Managers include at least one of the following modules: Activity Manager, used to interact with all running activities on the system; Location Manager, used to provide system services or applications with access to system location services; Package Manager, used to retrieve various information related to application packages currently installed on the device; Notification Manager, used to control the display and clearing of notification messages; and Window Manager, used to manage icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0091] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0092] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0093] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 100. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, the kernel layer may contain a driver layer, which is configured with hardware drivers. The driver layer may include at least one of the following hardware drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc. Specifically, when the display device 100 is only equipped with a front-facing camera 120a, the camera driver is used to drive the front-facing camera 120a.

[0094] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 100 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0095] In some embodiments, the display device 100 may install an application with shooting capabilities (hereinafter referred to as the target application), such as a camera application, a photo editing application, a QR code scanning application, or other applications that can access the camera. When the target application is already running at the application layer, the user can activate the shooting function, for example, by initiating QR code scanning through the target application's "Scan" entry, or by automatically activating the shooting function when the camera application is launched.

[0096] Figure 4 This is a schematic diagram of a shooting preview interface provided for some embodiments of this application.

[0097] See Figure 4After the target application starts the shooting function, it can display a shooting preview interface 40 on the monitor. The shooting preview interface 40 includes, but is not limited to: shooting preview window 41, shooting button 42 and flip button 43.

[0098] In some embodiments, the front-facing camera 120a of the display device 100 captures an original preview video stream (hereinafter referred to as the first preview stream), and the display device 100 obtains a processed second preview stream by performing image processing (including rotation processing) on ​​the preview stream, and displays the second preview stream in the shooting preview window 41.

[0099] In some embodiments, users can capture images by clicking the shutter button 42 or record videos by long-pressing the shutter button 42. The shooting preview interface 40 may include operation buttons corresponding to various shooting modes, including but not limited to: video, photo, portrait, panorama, beauty mode, cute mode, slow motion, etc.

[0100] In some embodiments, users can adjust the shooting direction by clicking the flip button 43, thus switching between the front and rear cameras. For example, if the target application is currently using the front camera, it switches to the rear camera in response to clicking the flip button 43. Some applications may enable the rear camera when the shooting function is activated; for example, the "scan" function uses the rear camera by default for scanning QR codes. Alternatively, an application may switch the target camera to the rear camera in response to triggering the flip button 43. However, if the display device 100 is only configured with a physical front camera 120a and does not have a physical rear camera (e.g., some tablets, electronic barcode scanners, etc.), the application cannot enable the rear camera, and the shooting preview interface 40 of the display device 100 will display a black screen or a message indicating that rear shooting is not supported, causing the application's shooting function to malfunction.

[0101] In cases where the display device 100 lacks a rear camera in its hardware, in order to meet the needs of the target application to use the rear camera for shooting, in some embodiments, the display device 100 can generate a virtual rear camera to simulate the shooting capabilities of a rear camera.

[0102] Figure 5 A schematic diagram illustrating the principle of generating a virtual camera for a display device 100 provided in some embodiments of this application.

[0103] In some embodiments, see Figure 5The controller 130 can run a driver layer and a HAL (Hardware Abstraction Layer). The driver layer can create a first video node by running relevant software modules (such as the V4l2 loopback module). The first video node is a virtual video node mapped by the rear camera simulated by the driver layer. A "video node" can be regarded as the storage space corresponding to the video device (specifically the camera) in memory. The video stream / image captured by the video device is imported into the video node and stored by the video node.

[0104] In some embodiments, see Figure 5 After creating the first video node, the driver layer can configure it to obtain its configuration information. This configuration information includes, but is not limited to, the first node address and the first node ID. The first node address is the storage address corresponding to the first video node, allowing access to the node and the retrieval of image data. The first node ID is used to identify the first video node. For example, the first node ID might be / dev / video1. It's important to note that the video node ID is unique; each video node has a different ID.

[0105] In some embodiments, the display device 100 has a physical front-facing camera 120a. When the driver layer detects that the front-facing camera 120a is connected, it can automatically create a second video node. This second video node is a real (non-virtual) video node mapped from the front-facing camera 120a. Similarly, see... Figure 5 After creating the second video node, the driver layer can configure it to obtain its configuration information. This configuration information includes, but is not limited to, the second node's address and its ID. The second node address is the storage address corresponding to the second video node, allowing access to the node and the retrieval of image data. The second node ID is used to identify the second video node. For example, the second node ID might be / dev / video2.

[0106] In some embodiments, see Figure 5 The driver layer can maintain a list of video nodes, which is used to maintain the configuration information of the created video nodes. The video node list includes at least the configuration information of the first node and the configuration information of the second node. After generating the virtual camera, the driver layer sends the video node list to the HAL layer.

[0107] In some embodiments, see Figure 5The HAL layer receives the list of video nodes and can enumerate the two logical cameras contained in the list. It sets the rear camera attribute (CAMERA_FACING_BACK) for the first video node mapped by the first node ID and sets the front camera attribute (CAMERA_FACING_FRONT) for the second video node mapped by the second node ID.

[0108] In some embodiments, see Figure 5 After setting the facing attribute of the two video nodes, the HAL layer assigns a first camera identifier to the first video node and a second camera identifier to the second video node. The first camera identifier indicates that the first video node is associated with a rear camera, and the second camera identifier indicates that the second video node is associated with a front camera. The representation of the first and second camera identifiers is not limited; for example, the first camera identifier can be "0" and the second camera identifier can be "1".

[0109] In some embodiments, see Figure 5 The HAL layer can maintain a camera list, which includes a mapping between first node configuration information and first camera identifiers, and a mapping between second node configuration information and second camera identifiers. In this way, the camera list can indicate the shooting capabilities of the display device 100. If the first camera identifier can be read from the camera list, it indicates that the display device 100 has rear camera shooting capabilities; if the second camera identifier can be read from the camera list, it indicates that the display device 100 has front camera shooting capabilities.

[0110] In some embodiments, see Figure 5 The HAL layer can send a list of cameras back to the driver layer. Upon receiving the camera list, the driver layer can determine that the first video node is associated with the virtual rear camera, and the second video node is associated with the physical front camera 120a. Since the display device 100 is configured with only one physical front camera 120a, and the virtual rear camera does not have real image capture capabilities, the driver layer can establish a virtual binding relationship between the first video node and the front camera. Thus, when the target application starts the rear camera, the driver layer can cache the preview stream captured by the front camera 120a to the first video node.

[0111] In some embodiments, in response to the activation of the shooting function, some applications can read the camera list of the HAL layer and query the shooting capabilities of the display device 100 through the camera list. When the target application can read the first camera identifier and the second camera identifier in the camera list, the target application cannot perceive that the rear camera is virtual, and therefore assumes that the display device 100 has a front camera and a rear camera, thereby selecting the target camera to be used.

[0112] In some embodiments, some applications, in response to the activation of the shooting function, may not read the camera list in the HAL layer or query the shooting capabilities of the display device 100, but instead directly activate the target camera. For example, if an application wants to activate the rear camera, since a virtual rear camera has already been created in the driver layer, and the front camera 120a assists the rear camera in image acquisition through a virtual binding relationship, the shooting preview interface will not display a black screen or a message indicating that rear shooting is not supported. In this way, by creating a virtual rear camera, the application's need to activate the rear shooting function is supported.

[0113] Figure 6 Comparison of preview effects of the front camera and rear camera provided in some embodiments of this application.

[0114] See Figure 6 If the display device 100 is equipped with both a physical front-facing camera 120a and a rear-facing camera 120c (not virtual), previewing and shooting can be performed according to the native Android Camera architecture. Before displaying the preview stream and photos, the target application can calculate the rotation angle based on the type of the enabled target camera (front-facing or rear-facing) and the current screen orientation, using the Orientation calculation formula configured in the Android system. This rotation angle represents the angle by which the original image captured by the target camera is rotated clockwise to match the current screen orientation.

[0115] The following section first introduces the key angles involved in previewing and shooting, including but not limited to: the installation angle of the gyroscope 120b, the screen rendering angle, and the camera imaging angle.

[0116] Figure 7 This is a schematic diagram showing the installation angle of a gyroscope provided in some embodiments of this application.

[0117] In some embodiments, after the gyroscope 120b is installed inside the display device 100 (i.e., the installation position is fixed), the installation angle of the gyroscope 120b is typically a fixed value. See also Figure 7With the display device 100 in portrait orientation and the front-facing camera 120a located on the right side (i.e., the camera is on the right side when the device is held vertically), the lines with solid black dots illustrate the gyroscope mounting direction. The rotation direction of the gyroscope 120b is clockwise, with the horizontal to the right as the rotation reference, so the mounting angle of the gyroscope 120b is 0°.

[0118] In some embodiments, the controller 130 runs a GPU (Graphics Processing Unit), which uses a screen rendering angle (denoted by degrees) when drawing and rendering images. The screen rendering angle is the angle between the GPU's image rendering direction and the screen orientation; that is, the screen rendering angle indicates from which position on the screen to begin rendering graphics. Therefore, there is a mapping relationship between the screen rendering angle and the screen orientation.

[0119] In some embodiments, when the screen orientation is portrait, the screen rendering angle mapped to the portrait orientation is 0°. When the screen orientation is landscape, the screen rendering angle mapped to the landscape orientation is 90°.

[0120] Figure 8 This is a schematic diagram illustrating the imaging angle of a camera provided in some embodiments of this application. Wherein, Figure 8 (a) in the example illustrates the imaging angle of the front-facing camera. Figure 8 (b) in the example illustrates the imaging angle of the rear camera.

[0121] In some embodiments, the imaging angle of a camera refers to the angle from which the camera begins exposure to the CMOS (Complementary Metal Oxide Semiconductor). Once the camera is mounted on the display device 100 (i.e., in a fixed position), the imaging angle of the camera is typically a fixed value.

[0122] In some embodiments, see Figure 8 In (a), the arrow illustrates the exposure direction of the front-facing camera 120a. With the horizontal to the right as the rotation reference and clockwise rotation as the positive direction, the imaging angle of the front-facing camera 120a is -90°.

[0123] In some embodiments, see Figure 8 In (b), the arrow illustrates the exposure direction of the rear camera 120c. With the horizontal to the right as the rotation reference and clockwise rotation as the positive direction, the imaging angle of the rear camera 120c is 90°.

[0124] In some embodiments, when the target application launches a preview of the target camera (front or rear camera), it can calculate the orientation (rotation angle). The standard formula for calculating the orientation (rotation angle) configured in the Android system is as follows:

[0125] Front-facing camera: Orientation = (camera_info.orientation + degrees) % 360 [Formula 1]

[0126] Orientation′=(360-Orientation)%360

Formula 2

[0127] Rear camera: Orientation = (camera_info.orientation - degrees + 360) % 360 [Formula 3]

[0128] Referring to Formulas 1 and 3, `camera_info.orientation` is the sensor orientation (sensor rotation angle) configured by the HAL layer and reported to the target application. `camera_info.orientation` represents the angle by which the gyroscope 120b rotates clockwise to match the camera's imaging angle. `degrees` represents the screen rendering angle. `%360` indicates a modulo operation, which normalizes the angle to a range of 0° to 360°, preventing overflow. `Orientation` represents the final calculated rotation angle.

[0129] Formula 1 allows us to calculate the orientation of the front-facing camera. Formula 2 then mirrors this orientation to obtain Orientation′. See also... Figure 6 , Figure 6 (a) in the example shows the user's original position. Figure 6 Example (c) shows the front preview position displayed in the preview interface after rotation processing based on Orientation′. Since the preview stream captured by the front camera has been mirrored, the front preview position is consistent with the user's original position.

[0130] Formula 3 provides the orientation for the rear camera. Figure 6Example (b) shows the rear preview position displayed in the preview interface after rotation processing based on Orientation. Since the preview stream captured by the rear camera is usually not mirrored, the rear preview position is a mirror image of the user's original position. For ease of description, this embodiment refers to the camera_info.orientation corresponding to the rear camera as the "first angle," which is the angle rotated clockwise to match the imaging angle of the rear camera (physical or virtual). This embodiment refers to the camera_info.orientation corresponding to the front camera 120a as the "second angle," which is the angle rotated clockwise to match the imaging angle of the front camera 120a.

[0131] Figure 9 This is a schematic diagram illustrating the calculation principle of the first and second angles provided for some embodiments of this application. Figure 9 (a) in the example illustrates the principle of calculating the first angle. Figure 9 Figure (b) illustrates the calculation principle of the second angle.

[0132] In some embodiments, see Figure 9 In (a), when rotating the mounting angle of the gyroscope 120b to match the imaging angle of the rear camera 120c, it needs to be rotated 90° clockwise; that is, the first angle is 90°. See also... Figure 9 In step (b), when rotating the mounting angle of the gyroscope 120b to match the imaging angle of the front-facing camera 120a, it needs to be rotated clockwise by 270°, i.e., the second angle is 270°. Therefore, for the display device 100 of the Android system, the camera_info.orientation corresponding to the front-facing camera 120a is usually set to 270°, and the camera_info.orientation corresponding to the rear-facing camera is usually set to 90°.

[0133] In some embodiments, if the display device 100 does not have a physical rear camera, but instead creates a virtual rear camera through the driver layer, the camera_info.orientation corresponding to the virtual rear camera can be set to 90°, that is, consistent with the camera_info.orientation corresponding to the physical rear camera.

[0134] In some embodiments, since the virtual rear camera does not have image acquisition capabilities, it still needs to rely on the front camera 120a to acquire preview streams and capture images. Therefore, the camera_info.orientation corresponding to the virtual rear camera can be set to 270°, that is, consistent with the camera_info.orientation corresponding to the physical front camera 120a.

[0135] In some embodiments, in addition to running the target application, the controller 130 may also run a preview processing module and a photo processing module. The preview processing module rotates the preview stream captured by the target camera and sends the rotated preview stream to the display so that the display shows the rotated preview stream on the photo preview interface. The photo processing module processes the images captured by the target camera, including generating images in a preset format and rotating the images.

[0136] In some embodiments, after the target application determines the type of the target camera to be used, the `camera_info.orientation` of the target camera is known. The target application then calculates the rotation angle according to the formula above and sends a shooting preview command to the preview processing module by calling the `setDisplayOrientation()` interface, instructing the preview processing module to adjust the preview direction. The shooting preview command includes the identifier of the target camera and the rotation angle (Orientation). If the target camera is a rear camera, the identifier is the first camera identifier; if the target camera is a front camera, the identifier is the second camera identifier.

[0137] In some embodiments, in response to a shooting preview command, if the target camera's identifier is identified as a first camera identifier (i.e., the target camera is a rear camera), the preview processing module can obtain the first node address from the HAL layer, access the first video node through the first node address, and obtain the first preview stream from the first video node. The preview processing module uses the rotation angle contained in the shooting preview command to rotate the preview stream.

[0138] In some embodiments, if the preview processing module identifies the target camera's identifier as a second camera identifier, i.e., the target camera is a front-facing camera, it can obtain the second node address from the HAL layer, access the second video node through the second node address, and obtain the preview stream from the second video node. The preview processing module uses the rotation angle contained in the shooting preview command to rotate the preview stream. Since the target application has already performed mirroring when calculating the rotation angle of the front-facing camera 120a (Formula 2), the preview processing module does not perform mirroring on the preview stream.

[0139] In some embodiments, during the process of displaying the preview stream on the shooting preview interface, refer to Figure 4 Users can trigger the camera button 42 at any time to capture an image. In response to this image capture operation, the target application can send a capture command to the image processing module. The capture command includes the target camera's identifier, a target timestamp, and a rotation angle calculated by the target application. The target timestamp represents the time when the target application received the image capture command.

[0140] In some embodiments, in response to a photo-taking command, if the target camera identifier is identified as a first camera identifier, i.e., the target camera is a rear camera, the photo-taking processing module can obtain the first node address from the HAL layer, access the first video node through the first node address, and obtain image frame data and thumbnail data corresponding to the target timestamp from the first video node, i.e., the image timestamps corresponding to the image frame data and thumbnail data are the same as the target timestamp.

[0141] In some embodiments, the first video node can store a preview stream captured by the target camera. The preview stream is essentially a real-time video stream, including multiple consecutive image frames, each image frame having a corresponding thumbnail. The image frames and their thumbnails have the same image timestamp. The image processing module can retrieve the image frames and thumbnails corresponding to the target timestamp from the preview stream cached by the first video node based on the target timestamp, thereby obtaining image frame data and thumbnail data.

[0142] In some embodiments, the image processing module acquires image frame data and thumbnail data, and can perform image processing. The image processing module processes the image frame data to generate an original captured image in a preset format, and processes the thumbnail data to generate an original thumbnail corresponding to the original captured image.

[0143] In some embodiments, the image processing module is, for example, a JPEGNode. The JPEGNode is a component running on the HAL layer, used to encode the YUV format image output by the camera sensor into JPEG format (a preset format). The JPEGNode includes submodules for setting the image rotation angle, specifically encodeMainJpeg and encodeThumnailJpeg. encodeMainJpeg rotates the original captured image according to the rotation angle contained in the image capture command to obtain the target main image. encodeThumnailJpeg rotates the original thumbnail according to the rotation angle contained in the image capture command to obtain the target thumbnail. The only difference between the target main image and the target thumbnail is the image size; the target thumbnail is equivalent to a scaled-down version of the target subject.

[0144] In the case where the display device 100 only has a physical front-facing camera 120a and creates a virtual rear-facing camera through the driver layer, if the target application's intended camera is the rear-facing camera, the preview processing module may cause image inversion when rotating the preview stream based on the rotation angle issued by the target application in response to the shooting preview command. Similarly, the image inversion problem may also occur when the photo-taking processing module rotates the original captured image and the original thumbnail based on the rotation angle issued by the target application in response to the photo-taking command. The reasons for this image inversion are explained below:

[0145] Figure 10 This is a schematic diagram illustrating image inversion when using a rear-facing camera in some embodiments of this application. Figure 10 In example (a), in the user's original position in a landscape shooting scenario, since the virtual rear camera does not have shooting capabilities, the display device 100 still needs to use the front camera 120a for previewing and shooting. Therefore, the display device 100 is placed in landscape mode, and the user stands in front of the screen, facing the front camera 120a. Figure 10 (b) in the image shows the original image taken by the front camera 120a (before rotation) based on the user's original position in a landscape shooting scenario. Figure 10 (c) shows the display effect after rotating the original image captured by the front-facing camera 120a.

[0146] If the camera_info.orientation configured for the rear camera in the HAL layer is 270°, and the display device 100 is in landscape mode (degrees 90°), then the target application calculates Orientation based on formula 3 above: Orientation = (270° - 90° + 360) % 360 = 180°. Therefore, Figure 10 The final preview effect of the original image in (b) after rotating it 180° clockwise is as follows: Figure 10 As shown in (c) in the figure, by comparison Figure 10 As can be seen in (a) and (c), the image displayed in the shooting preview interface is inverted compared to the user's original position.

[0147] Figure 11 Illustrations of image inversion occurring when the target application using the rear camera, provided in some embodiments of this application. Figure Two .in, Figure 11In example (a) of the example, in the user's original position in the vertical shooting scenario, since the virtual rear camera does not have shooting capability, the display device 100 still needs to use the front camera 120a for previewing and shooting. Therefore, the display device 100 is placed vertically, and the user stands in front of the screen, facing the front camera 120a. Figure 11 (b) is the original image effect (before rotation) captured by the front camera 120a based on the user's original position in a portrait shooting scenario; Figure 11 (c) shows the display effect after rotating the original image captured by the front-facing camera 120a.

[0148] If the camera_info.orientation configured for the rear camera in the HAL layer is 90°, and the display device 100 is in portrait mode (degrees are 0°), then the target application calculates Orientation based on formula 3 above: Orientation = (90° - 0° + 360) % 360 = 90°. Therefore, Figure 11 The final image displayed on the preview interface after rotating the original image in (b) 90° clockwise is as follows: Figure 11 As shown in (c) in the figure, by comparison Figure 11 As can be seen in (a) and (c), the image displayed in the shooting preview interface is inverted compared to the user's original position.

[0149] In other words, if the target application enables the rear camera, and the rear camera is a virtual camera created by the driver layer, and the `camera_info.orientation` corresponding to the rear camera configured in the HAL layer is 270°, then the preview and shooting will result in inverted images when the display device 100 is in landscape mode. If the `camera_info.orientation` corresponding to the rear camera configured in the HAL layer is 90°, then the preview and shooting will result in inverted images when the display device 100 is in portrait mode. Since the front camera 120a is a physical camera, when the target application enables the front camera 120a, the preview and shooting will not result in inverted images regardless of whether the display device 100 is in landscape or portrait mode. If the display device 100 is configured with a physical rear camera 120c, then the image inversion problem will not occur when shooting with the rear camera, regardless of whether the screen is in landscape or portrait mode.

[0150] Figure 12 A flowchart of an image processing method provided in some embodiments of this application.

[0151] For a scenario where the display device 100 is configured with only one physical front-facing camera 120a, and the rear-facing camera is a virtual camera created by the driver layer, to address the image inversion issue that occurs when a target application with shooting capabilities uses the rear-facing camera for previewing, the controller 130 runs a target application, a driver layer, and a preview processing module. The target application specifically runs at the application layer, and the preview processing module specifically runs at the HAL layer. Figure 12 The methods provided include:

[0152] Step S121: The target application receives the operation to start the shooting function and determines the target camera to be used by the target application.

[0153] In some embodiments, users can initiate the shooting function through a specific function entry (such as "scan") after the target application is launched. If the target application is a camera application, such applications typically enable shooting preview upon launch, so launching the target application is equivalent to launching the shooting function.

[0154] In some embodiments, the target camera is one of a front-facing camera and a rear-facing camera. The target application reads the camera list by accessing the HAL layer. As long as the first camera identifier is read, the target application assumes that the display device 100 has rear-facing shooting capabilities, and the target application cannot perceive that the rear-facing camera is a virtual camera.

[0155] In step S122, when the target application determines that the target camera is a rear camera, it sends a first shooting preview command to the preview processing module.

[0156] The first shooting preview command includes a first camera identifier corresponding to the rear camera, and may not include the rotation angle calculated by the target application using a formula. The camera identifier is equivalent to the camera's ID, used to identify the camera category; the first camera identifier maps to a virtual rear camera. The target application enables the rear camera by including the first camera identifier in the shooting preview command.

[0157] In step S123, the preview processing module responds to the first shooting preview command by sending a command to start the target camera to the driver layer.

[0158] In some embodiments, when the preview processing module receives the first shooting preview instruction, it needs to notify the driver layer to start the target camera in order to acquire the preview stream; on the other hand, it needs to acquire the rotation angle in order to perform rotation processing on the preview stream.

[0159] In some embodiments, the preview processing module may send a start target camera instruction to the driver layer to instruct the driver layer to start the target camera. The start target camera instruction may include a first camera identifier corresponding to the rear camera.

[0160] In step S124, the driver layer responds to the command to start the target camera, starts the front camera, and stores the first preview stream captured by the front camera into the first video node.

[0161] In some embodiments, when the driver layer receives a command to start the target camera, it determines that the target camera to be started is a rear camera. Since the rear camera is a virtual camera and does not have real image acquisition capabilities, the driver layer needs to start the front camera 120a, which will then capture a first preview stream. The first preview stream is the raw preview stream captured by the front camera 120a, which has not yet undergone rotation processing.

[0162] In some embodiments, the driver layer first establishes a virtual binding relationship between the front-facing camera 120a and the first video node. Thus, when the target application uses the rear camera, the driver layer activates the front-facing camera 120a and stores the first preview stream captured by the front-facing camera 120a in the first video node, thereby simulating the rear camera capturing a preview stream. It should be noted that at the user level, the user is aware that the display device 100 does not have a physical rear camera; therefore, after activating the shooting function of the target application, the user will actively use the front-facing camera 120a for previewing and taking photos.

[0163] In step S125, the preview processing module obtains the first rotation angle based on the first camera identifier and the current screen orientation of the display device.

[0164] Step S126: The preview processing module obtains the first preview stream from the first video node.

[0165] Step S127: Rotate the first preview stream according to the first rotation angle to obtain the second preview stream, and transmit the second preview stream to the display.

[0166] In step S128, the monitor displays the second preview stream on the shooting preview interface.

[0167] As analyzed in the preceding embodiments, the rotation angle is related to the camera type and screen orientation. The preview processing module no longer processes the preview stream based on the rotation angle calculated and issued by the target application using the aforementioned formula. Instead, it automatically matches the rotation angle using the first camera identifier and the current screen orientation to prevent image inversion after processing the preview stream with this rotation angle. This application refers to the rotation angle matched by the preview processing module as the first rotation angle.

[0168] In some embodiments, the preview processing module detects that the first shooting preview command contains a first camera identifier, and knows that the target camera is a rear camera (virtual). In this case, the first preview stream is captured by the front camera 120a. See [link to relevant documentation]. Figure 10 and Figure 11The image may be upside down in the shooting preview interface. Therefore, the preview processing module needs to correct the rotation angle, that is, to use the matched first rotation angle to rotate the first preview stream, thereby avoiding image inversion.

[0169] Figure 13 A schematic diagram illustrating how image inversion is eliminated by correcting the rotation angle when using a virtual rear camera in a target application provided in some embodiments of this application.

[0170] contrast Figure 10 and Figure 13 When the screen orientation is landscape, Figure 10 The image is rotated 180° clockwise by the target application using a rotation angle (180°), resulting in an image inversion issue. The preview processing module addresses this by removing the image inversion. (See [link to relevant documentation]). Figure 13 The first rotation angle for matching is 0°, thus rotating the original image clockwise by 0° (i.e., without rotation). The final image displayed on the shooting preview interface is as follows. Figure 13 As shown in (c) in the figure, by comparison Figure 13 As shown in (a) and (c), the preview stream image displayed in the shooting preview interface does not have an inverted problem.

[0171] Figure 14 This illustration shows how image inversion is eliminated by correcting the rotation angle when using a virtual rear camera in target applications provided in some embodiments of this application. Figure Two .

[0172] contrast Figure 11 and Figure 14 When the screen orientation is portrait, Figure 11 The image is rotated 90° clockwise by the target application using a specified rotation angle (90°), resulting in an inverted image. The preview processing module addresses this by removing the inverted image. (See [link to relevant documentation]). Figure 14 The first rotation angle matched is 270°, thus rotating the original image 270° clockwise, resulting in the image displayed in the shooting preview interface as shown below. Figure 14 As shown in (c) in the figure, by comparison Figure 14 As shown in (a) and (c), the preview stream image displayed in the shooting preview interface does not have an inverted problem.

[0173] Although this application uses the front-facing camera 120a to capture the preview stream, the target application actually uses the rear-facing camera, so the preview processing module does not need to mirror the first preview stream.

[0174] In some embodiments, the preview processing module can obtain the address of a first node from the camera list maintained by the HAL layer, access the first video node through the first node address, and obtain the first preview stream from the first video node. The first preview stream is a video stream formed by multiple consecutive frames of original images. Each frame of the original image can share a first rotation angle, that is, the preview processing module can rotate the first preview stream based on the first rotation angle, that is, rotate each frame of the original image included in the first preview stream, so that the second preview stream displayed on the shooting preview interface will not have the problem of image inversion.

[0175] In some embodiments, the Android system's Camera architecture can set and store a preset map by running the OrientationUtil (orientation utility class). This preset map stores the mapping relationship between shooting scene information and rotation angles, thereby specifying the correct rotation angle to use in different shooting scenarios and preventing abnormalities such as inverted images. The shooting scene information includes the camera identifier and screen orientation. For example, in the preset map, {first camera identifier, landscape} maps to a rotation angle of 0°; and {first camera identifier, portrait} maps to a rotation angle of 270°.

[0176] In some embodiments, in response to a first shooting preview command, the preview processing module uses {first camera identifier, current screen orientation} as a key and retrieves the value mapped to that key from a preset mapping table. This value is the first rotation angle obtained by the preview processing module. By setting a preset mapping table, the preview processing module can quickly query the first rotation angle from the map table based on preset mapping rules, thus ensuring that the rotation angle conforms to the conditions of the current shooting scene, ensuring the accuracy of the shooting preview, avoiding problems such as image inversion caused by abnormal rotation angles, and eliminating the need for the preview processing module to dynamically match the first rotation angle, thereby improving the efficiency of rotation angle acquisition and image processing.

[0177] In some embodiments, the camera orientation may be handled differently for different applications (especially third-party applications), which may prevent different applications from using the same preset mapping table. To address this, when setting the preset mapping table, the shooting scene information also includes the application identifier of the application with shooting functionality. Thus, in response to the first shooting preview command, the preprocessing module uses {the target application's application identifier, the first camera identifier, and the current screen orientation} as a key to retrieve the first rotation angle mapped to that key from the preset mapping table, thereby ensuring that the first rotation angle meets the camera processing requirements of the target application.

[0178] In some embodiments, the application identifier includes, but is not limited to, an application ID and an application package name. Both the application ID and the application package name are unique and can accurately identify the category of the application.

[0179] In some embodiments, some applications only support portrait shooting mode. When the display device 100 switches from portrait to landscape mode, the image displayed in the shooting preview interface will not rotate synchronously, resulting in the image in the shooting preview interface not being displayed upright in landscape mode, which affects the preview and shooting effect.

[0180] In some embodiments, the display device may have at least one of an automatic rotation function and a forced landscape function. When the automatic rotation function is enabled, the preview processing module will capture an image displayed on the preview interface and rotate it synchronously with the screen, thereby ensuring that the preview image and screen orientation remain synchronized and consistent. If the forced landscape function is enabled, regardless of whether the current screen orientation is landscape or portrait, the preview processing module will forcibly rotate the image displayed on the preview interface to the landscape orientation. The forced landscape function may cause the following problems:

[0181] Issue a: Preview orientation is incorrect. Figure 15 This is a schematic diagram showing the preview interface captured after the forced landscape mode function is enabled when the display device 100 provided in some embodiments of this application is in portrait mode. See also Figure 15 When the display device 100 is in portrait mode, if an application enables the front-facing camera 120a and forces the screen to be in landscape mode, the image displayed in the shooting preview interface (hereinafter referred to as the preview image) will appear to have the head facing to the right compared to the original position (it will not be displayed upright).

[0182] Issue b: Preview aspect ratio misalignment. The forced landscape mode may alter the original aspect ratio of the preview image, causing it to be stretched, compressed, or cropped, thus affecting the preview quality.

[0183] To address issues a and b, the abnormal preview orientation and distorted preview ratio can be eliminated by adjusting the rotation angle of the preview image. To this end, when setting the preset mapping table, the shooting scene information also includes the functional status of the display device 100. The functional status of the display device 100 includes at least one of the on / off states of the automatic rotation function and the forced landscape function. Responding to the first shooting preview command, the preprocessing module uses {the application identifier of the target application, the first camera identifier, the current screen orientation, and the current functional status of the display device} as a key to retrieve the first rotation angle mapped to that key in the preset mapping table.

[0184] In some embodiments, the current functional state of the display device 100 can be automatic rotation function on and forced landscape function off. Here, "on" indicates an enabled state, and "off" indicates a disabled state. If the display device 100 has both automatic rotation and forced landscape functions, typically, these two functions are not enabled simultaneously.

[0185] In some embodiments, the current functional state of the display device 100 can be forced landscape mode on and automatic rotation mode off.

[0186] In some embodiments, the current functional state of the display device 100 can be either auto-rotation on or auto-rotation off. In this embodiment, the display device 100 does not have a forced landscape mode function.

[0187] In some embodiments, the current functional state of the display device 100 can be either forced landscape mode on or forced landscape mode off. In this embodiment, the display device 100 does not have an automatic rotation function. In the embodiments of the above image processing method, for display devices that are only configured with a front-facing camera 120a, by creating a virtual rear-facing camera and having the front-facing camera 120a assist the rear-facing camera in image acquisition, the problem of black screen during rear-facing shooting caused by hardware configuration deficiencies can be solved. This allows applications to enable rear-facing shooting and avoids the abnormal rotation angle caused by the virtual rear-facing camera lacking image acquisition capabilities and the front-facing camera still being responsible for image acquisition when the rear-facing camera is enabled. This prevents image inversion and ensures the accuracy of the shooting preview interface display. By creating a preset mapping table, the first rotation angle for different shooting scenarios can be specified in the preset mapping table, ultimately providing a more complete and accurate mapping rule. This ensures that the rotation angle conforms to the conditions of the current shooting scenario, avoiding image inversion caused by abnormal rotation angles, as well as abnormal preview orientation and preview aspect ratio issues caused by forced landscape rotation. At the same time, it can adapt to the camera processing methods of different applications (especially third-party applications), ensuring the accuracy of shooting preview and improving the display effect of the shooting preview interface.

[0188] In the above embodiments, the target application may not calculate the rotation angle corresponding to the rear camera (hereinafter referred to as: the second rotation angle) based on Formula 3, and the first shooting preview instruction does not include the second rotation angle. In this way, the preview processing module can automatically match the first rotation angle based on the first camera identifier and the current screen orientation. However, this method requires modification of the target application's original software program so that the target application does not execute the program for calculating and issuing the second rotation angle.

[0189] Figure 16 Flowchart of the image processing method provided in some embodiments of this application Figure Two .

[0190] In some embodiments, see Figure 16 When the target application determines that the target camera to be used is a rear camera (virtual), it can obtain a first angle from the HAL layer. The first angle is the `camera_info.orientation` corresponding to the rear camera, and the first angle can be 270° or 90°. The target application determines the screen rendering angle `degrees` based on the current screen orientation; the screen rendering angle mapped to the portrait orientation is 0°, and the screen rendering angle mapped to the landscape orientation is 90°. Thus, the target application can calculate a second rotation angle based on formula 3, i.e., `Orientation = (camera_info.orientation - degrees + 360) % 360`. The second rotation angle is the `Orientation` calculated by the target application. See also the following examples in some embodiments: Figure 16 The target application sends a first shooting preview instruction to the preview processing module. The first shooting preview instruction includes a second rotation angle and a first camera identifier corresponding to the rear camera.

[0191] In some embodiments, see Figure 16 Upon receiving the first shooting preview instruction, the preview processing module sends a target camera activation instruction to the driver layer. This target camera activation instruction may contain a first camera identifier corresponding to the rear camera. In response to the target camera activation instruction, the driver layer reads the first camera identifier contained in the instruction. If it confirms that the target application wants to enable the rear camera, the driver layer actually activates the front camera 120a and stores the first preview stream captured by the front camera 120a in the first video node. The first video node is a virtual video node created earlier by the driver layer and mapped to the rear camera.

[0192] In some embodiments, see Figure 16 The preview processing module can dynamically match the first rotation angle, or query the first rotation angle from a preset mapping table, and compare the first rotation angle with the second rotation angle.

[0193] In some embodiments, see Figure 16 If the second rotation angle differs from the first rotation angle, it indicates that the second rotation angle calculated and issued by the target application is incorrect. The preview processing module then needs to correct the rotation angle to prevent the preview image from being inverted and to improve the accuracy of the shooting preview interface. The preview processing module can obtain the camera list from the HAL layer and obtain the first node address mapped to the first camera identifier. This allows the preview processing module to access the first video node through the first node address and obtain the first preview stream from the first video node.

[0194] In some embodiments, seeFigure 16 The preview processing module can replace the second rotation angle with the first rotation angle, that is, perform rotation processing on the first preview stream based on the first rotation angle. In this way, the target application still calculates and sends the second rotation angle according to the original logic, without changing the target application's software program configuration. Subsequently, the preview processing module determines whether the rotation angle sent by the target application needs to be corrected by comparing the first rotation angle and the second rotation angle.

[0195] In some embodiments, if the target application enables a virtual rear camera, referring to the analysis of the foregoing embodiments, the second rotation angle and the first rotation angle must be different, which will cause the preview image to be inverted. Therefore, the preview processing module needs to correct the rotation angle of the preview image.

[0196] Figure 17 Flowchart of the image processing method provided in some embodiments of this application Figure Three .

[0197] In some embodiments, see Figure 17 When the target application determines that the target camera to be used is the front-facing camera (physical), it can obtain a second angle from the HAL layer. The second angle is the `camera_info.orientation` corresponding to the front-facing camera, and it is usually set to 270°. The target application determines the screen rendering angle `degrees` based on the current screen orientation: 0° for portrait orientation and 90° for landscape orientation. Thus, the target application can calculate the rotation angle `Orientation` before mirroring based on Formula 1, i.e., `Orientation = (camera_info.orientation + degrees) % 360`, and then continue to mirror the `Orientation` based on Formula 2 to obtain the rotation angle `Orientation′` after mirroring, where `Orientation′` is the third rotation angle.

[0198] In some embodiments, see Figure 17 The target application sends a second shooting preview instruction to the preview processing module. The second shooting preview instruction includes a third rotation angle and a second camera identifier corresponding to the front camera.

[0199] In some embodiments, see Figure 17In response to the second shooting preview command, the preview processing module sends a target camera activation command to the driver layer. This target camera activation command may contain a second camera identifier corresponding to the front-facing camera. In response to the target camera activation command, the driver layer reads the second camera identifier contained in the command, confirms that the target application wants to enable the front-facing camera, activates the front-facing camera 120a, and stores the first preview stream captured by the front-facing camera 120a in the second video node. The second video node is the real video node mapped by the front-facing camera 120a.

[0200] In some embodiments, see Figure 17 The preview processing module can obtain the camera list from the HAL layer and obtain the second node address mapped by the second camera identifier. In this way, the preview processing module can access the second video node through the second node address and obtain the first preview stream from the second video node.

[0201] In some embodiments, since the front-facing camera 120a is a real physical camera, not a virtual camera, the third rotation angle calculated by the target application using Formulas 1 and 2 is accurate and will not cause the preview image to be inverted. Therefore, see Figure 17 When the target application enables the front-facing camera 120a, the preview processing module does not need to obtain the first rotation angle. Instead, it directly rotates the first preview stream based on the third rotation angle to obtain the third preview stream, and sends the third preview stream to the display, which then displays the third preview stream on the shooting preview interface.

[0202] In this way, the target application still calculates and issues the third rotation angle according to its original logic, without requiring any changes to the target application's software configuration. Subsequently, the preview processing module processes the preview stream based on the third rotation angle. The preview processing module does not need to dynamically set the first rotation angle, nor does it need to retrieve the first rotation angle from a preset mapping table. Instead, it directly uses the third rotation angle issued by the target application to process the preview stream, thereby reducing the number of procedures executed by the preview processing module and improving image processing efficiency while ensuring the accuracy of the shooting preview.

[0203] Figure 18 Flowchart of the image processing method provided in some embodiments of this application Figure Four .

[0204] In the case where the display device 100 is configured with only one physical front camera 120a and the rear camera is a virtual camera created by the driver layer, similar to the shooting preview, in order to solve the image inversion problem that occurs when the target application with shooting function takes a picture using the rear camera, the controller 130 runs a target application, a driver layer and a picture processing module, wherein the target application specifically runs in the application layer and the picture processing module (e.g. JPEGNode) specifically runs in the HAL layer.

[0205] In some embodiments, see Figure 18 During the process of displaying the rotated preview stream in the shooting preview interface, the user can trigger... Figure 4 The example shows a camera button 42 for inputting an image capture operation. In response to this image capture operation, the target application detects that the current camera is a rear camera and sends a capture command to the image processing module. This capture command includes a first camera identifier corresponding to the rear camera and a target timestamp. The target timestamp represents the time when the target application receives the image capture operation, and then accurately determines the image frame to be processed.

[0206] In some embodiments, in response to a photo-taking command, the image processing module recognizes that the command contains a first camera identifier. It can then obtain the first node address from the HAL layer, access the first video node through the first node address, and retrieve image frame data and thumbnail data corresponding to the target timestamp from the first video node. That is, the image timestamps corresponding to the image frame data and thumbnail data are the same as the target timestamp. The image timestamp is a timestamp set by the front-facing camera 120a for the image frame, and it is used to characterize the time when the image frame was captured by the front-facing camera 120a.

[0207] In some embodiments, the first video node may store a first preview stream captured by the front-facing camera 120a. The first preview stream is essentially a real-time video stream, including multiple consecutive image frames, each image frame having a corresponding thumbnail. The image frames and their thumbnails have the same image timestamp. The image processing module may, based on a target timestamp, retrieve image frames and thumbnails matching the target timestamp from the first preview stream cached by the first video node, thereby obtaining image frame data and thumbnail data.

[0208] In some embodiments, the first video node can simultaneously maintain a first preview stream, a first image capture stream, and a second image capture stream. The first image capture stream is the stream of images captured by the front-facing camera 120a corresponding to the main image, and the second image capture stream is the stream of images captured by the front-facing camera 120a corresponding to thumbnails. Thus, the image processing module obtains the first image capture stream, captures image frames corresponding to the target timestamp from the first image capture stream, and obtains image frame data. The image processing module obtains the second image capture stream, captures thumbnails corresponding to the target timestamp from the second image capture stream, and obtains thumbnail data.

[0209] In some embodiments, see Figure 18 The image processing module processes the image frame data to generate a raw image in a preset format, and rotates the raw image based on a first rotation angle to obtain the target main image.

[0210] In some embodiments, seeFigure 18 The image processing module processes the thumbnail data to generate an original thumbnail corresponding to the original captured image, and rotates the original thumbnail based on the first rotation angle to obtain the target thumbnail.

[0211] In some embodiments, the image processing module can dynamically set the first rotation angle, or retrieve a matching first rotation angle from a preset mapping table. For details, refer to the method by which the preview processing module obtains the first rotation angle.

[0212] In some embodiments, since the preview processing module has already obtained the first rotation angle during the preview stage, the image processing module can directly obtain the first rotation angle from the preview processing module. This eliminates the need for the image processing module to dynamically set the first rotation angle or retrieve a matching first rotation angle from a preset mapping table, thus improving the efficiency of obtaining the rotation angle and enhancing image processing efficiency.

[0213] In some embodiments, the image processing module is, for example, a JPEGNode. The JPEGNode is a component running on the HAL layer, used to encode the YUV format image output by the camera sensor into JPEG format (a preset format). The JPEGNode includes sub-modules for image rotation processing, specifically encodeMainJpeg and encodeThumnailJpeg. encodeMainJpeg rotates the original captured image according to a first rotation angle to obtain the target main image. encodeThumnailJpeg rotates the original thumbnail according to the first rotation angle to obtain the target thumbnail.

[0214] Both previewing and taking photos require image rotation, typically using the same rotation angle. This means that when the target application uses the rear camera, not only will the image be inverted due to abnormal rotation angles during previewing, but the main image and thumbnail will also be inverted when the user clicks the shutter button. To address this, this application obtains image frame data and thumbnail data corresponding to the target timestamp from a virtual video node during the photo-taking stage. The photo-taking processing module rotates the image frames and thumbnails separately based on a first rotation angle to obtain the target main image and target thumbnail. This ensures that both the main image and its corresponding thumbnail are processed using the correct rotation angle, avoiding image inversion caused by incorrect rotation angles and improving the accuracy of the shooting angle and image display effect.

[0215] In some embodiments, see Figure 18The image processing module can also transmit the target main image and target thumbnail to the display, so that the display shows the target main image and / or target thumbnail in the user interface based on the target application's shooting function configuration. Different applications may have different shooting function configurations, and the display effect of the user interface when the photo is taken may vary.

[0216] Figure 19 This is a schematic diagram of the user interface when taking a photo is completed, provided for some embodiments of this application.

[0217] In some embodiments, upon completion of shooting, see Figure 19 The user interface can still display the shooting preview interface, and show the target thumbnail in a preset position (such as the lower left corner) on the shooting preview interface. In this way, the target application can support users to shoot continuously, improving the shooting speed when taking multiple pictures in succession. When the user clicks on the target thumbnail, the controller can display the target main image corresponding to the target thumbnail on the user interface, so that the user can easily view the large image.

[0218] Figure 20 A schematic diagram of the user interface when taking a photo is completed, provided for some embodiments of this application. Figure Two .

[0219] In some embodiments, upon completion of shooting, see Figure 20 The user interface can display a shooting confirmation page, which includes, but is not limited to, the target main image, a return button, a save button, and an edit button.

[0220] In some embodiments, users can view the shooting effect by taking a picture of the target main image displayed on the confirmation page. If the user is not satisfied with the current shooting effect, they can click the back button. In response to clicking the back button, the target application controls the display to return to the shooting preview interface so that the user can preview and take a picture again. In this scenario, the target main image and target thumbnail are not saved.

[0221] In some embodiments, if the user is satisfied with the current shooting effect, they can click the save button. In response to clicking the save button, the target application can store the target main image and the target thumbnail in the memory of the display device 100, and control the display to return to the shooting preview interface so that the user can continue previewing and taking photos. The user can also view thumbnails of the captured images through the photo album application; clicking any thumbnail A will allow the user to view the corresponding main image A.

[0222] In some embodiments, in response to clicking the image editing button, the target application can activate its image editing function and control the display to show the image editing operation page, so that the user can perform beautification, beauty enhancement, filter adjustment, and other retouching processes on the target main image.

[0223] Figure 21 Flowchart of the image processing method provided in some embodiments of this application Figure Five . Figure 21 The example illustrates the interactive processing flow from image processing of the first preview stream to displaying the second preview stream on the capture preview interface, including rotating the first preview stream based on a rotation angle (first rotation angle, second rotation angle, or third rotation angle).

[0224] In some embodiments, see Figure 21 The display device 100 can run a target application, a framework layer, a HAL layer, a driver layer, camera hardware, and SurfaceFlinger (display compositor). The framework layer runs a camera service (CameraService). The camera hardware includes a camera sensor and an ISP (Image Signal Processor). SurfaceFlinger is a core graphics display management component in the Android system, responsible for combining the graphical content generated by different applications into the final screen image.

[0225] In some embodiments, see Figure 21 The target application can send a preview request to the CameraService in the framework layer. The preview request includes `createCaptureSession()` and `setRepeatingRequest()`. `createCaptureSession()` is the core function of the Android Camera API used to create a CaptureSession. By configuring the output stream (including the preview stream and the photo stream) and callback mechanism, it establishes an interaction channel between the target application and the camera device, enabling functions such as previewing, taking photos, and recording videos. `setRepeatingRequest()` sets up repeated requests to achieve continuous previewing, rather than a single photo capture. The preview request can also include the identifier of the target camera to be enabled and the rotation angle A. If the target application enables the rear camera, the identifier is the first camera identifier; if the target application enables the front camera, the identifier is the second camera identifier. The rotation angle A is the rotation angle calculated by the target application, which can be either the second rotation angle (rear camera) or the third rotation angle (front camera).

[0226] In some embodiments, see Figure 21The framework layer runs CameraService, an Android system-level service responsible for managing camera resources and request scheduling. CameraService responds to preview requests and can configure the output stream (such as preview stream, photo stream, etc.) using configureStream(), and allocate a BufferQueue (a buffer queue used to transfer image data) for the output stream.

[0227] In some embodiments, CameraService can obtain rotation angle B based on the identifier of the target camera included in the preview request and the current screen orientation. Rotation angle B is a rotation angle that CameraService automatically matches or retrieves from a preset mapping table. CameraService determines the final correct target rotation angle by comparing rotation angle A and rotation angle B, and sends the target rotation angle to the HAL layer.

[0228] In some embodiments, see Figure 21 The HAL layer is an "abstraction layer" between the system and the hardware, responsible for translating requests from the framework layer into instructions that the hardware can understand. The HAL layer uses ioctl(VIDIOC_S_FMT) to set the camera hardware's resolution, format (e.g., YUV420, RGB), thereby completing the hardware parameter configuration.

[0229] In some embodiments, see Figure 21 The driver layer acts as a "bridge" between the hardware and system software, responsible for driving and controlling the camera hardware. It initializes the camera sensor (responsible for capturing raw image data) and initiates the ISP module to perform image processing such as noise reduction, white balance, and color correction.

[0230] In some embodiments, see Figure 21 The camera hardware acts as the "data producer," responsible for capturing and processing images. After driver initialization, the camera sensor sends a capture-ready interrupt message to the driver layer upon capturing a frame, notifying the driver layer that the preview stream acquisition is ready. Afterward, the camera sensor can continuously acquire images to form a preview stream.

[0231] In some embodiments, see Figure 21The driver layer can use DMA (Direct Memory Access, which allows for efficient data transfer bypassing the CPU) to transfer the first preview stream captured by the camera sensor to the Gralloc Buffer. The Gralloc Buffer is a memory buffer in the Android graphics system used by the CPU and GPU to share graphics data. The Gralloc Buffer can contain a first video node and a second video node created by the driver layer. When the target application enables the rear camera, the driver layer can cache the first preview stream to the first video node via DMA. When the target application enables the front camera, the driver layer can cache the first preview stream to the second video node via DMA.

[0232] In some embodiments, see Figure 21 The HAL layer processes the original image of the first preview stream, including but not limited to rotation, scaling, and format conversion, and finally generates a second preview stream suitable for display. The HAL layer can rotate the first preview stream based on the target rotation angle provided by CameraService to avoid display abnormalities such as image inversion, abnormal preview orientation, and preview scale misalignment.

[0233] In some embodiments, see Figure 21 The HAL layer can send an onFrameAvailable() callback to the CameraService. The onFrameAvailable() callback is used to notify that frame data is available, indicating that the second preview stream is ready and waiting to be displayed.

[0234] In some embodiments, see Figure 21 When CameraService receives the onFrameAvailable() callback, it writes the image data of the second preview stream to the SurfaceFlinger's Surface (input layer) using the queueBuffer() method. Then, SurfaceFlinger composites all Surface layers (including the preview layer, UI layer, etc.) and finally outputs the result to the display, thereby refreshing the target application's interface and completing the camera preview.

[0235] The preview processing module in the above embodiments is equivalent to a combination of the CameraService and the HAL layer modules that perform graphics processing on the first preview stream. The system framework and interaction logic of the display device 100 are not limited to the embodiments of this application.

[0236] In some embodiments, a computer storage medium is also provided, which may store a program. When the computer storage medium is configured in the display device 200, the program, when executed, may include the program steps involved in the image processing methods of the above embodiments. The computer storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0238] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A display device, characterized in that, include: A front-facing camera, configured to capture images from one side of the screen; The display is configured to show the shooting preview interface; A controller, coupled to the display and the front-facing camera, runs a preview processing module, a driver layer, and a target application, the target application having a shooting function. The controller is configured to: In response to receiving an operation to activate the shooting function of the target application, the target camera to be used by the target application is determined; the target camera is one of the front camera and the rear camera, and the rear camera is a virtual camera generated by the driver layer by creating a first video node, the first video node being a virtual video node associated with the rear camera; When the target camera is determined to be the rear camera, a first shooting preview instruction is sent to the preview processing module through the target application. The first shooting preview instruction includes a first camera identifier corresponding to the rear camera. In response to the first shooting preview command, the preview processing module sends a command to start the target camera to the driver layer. In response to the command to start the target camera, the driver layer starts the front camera and stores the first preview stream captured by the front camera into the first video node. The preview processing module obtains a first rotation angle based on the first camera identifier and the current screen orientation of the display device. The first rotation angle is a rotation angle matched by the preview processing module. The rotation angle is the angle at which the original image captured by the front camera is rotated clockwise to be consistent with the current screen orientation. The preview processing module obtains a first preview stream from the first video node, rotates the first preview stream according to the first rotation angle to obtain a second preview stream, and transmits the second preview stream to the display so that the display shows the second preview stream on the shooting preview interface.

2. The display device according to claim 1, characterized in that, The display device also includes a gyroscope and a graphics processor; Before sending a first shooting preview command to the preview processing module by determining the target camera to be used by the target application, the controller is also configured to: When the target camera is determined to be the rear camera, a first angle is obtained through the target application. The first angle is the angle at which the mounting angle of the gyroscope is rotated clockwise to match the imaging angle of the rear camera. The target application obtains the screen rendering angle mapped to the current screen orientation of the display device, where the screen rendering angle is the angle between the image rendering direction of the graphics processor and the screen orientation. The target application calculates the second rotation angle based on the first angle and the screen rendering angle; The first shooting preview instruction is generated by the target application, and the first shooting preview instruction also includes the second rotation angle.

3. The display device according to claim 2, characterized in that, The preview processing module obtains a first preview stream from the first video node, rotates the first preview stream according to the first rotation angle, and obtains a second preview stream, specifically configured as follows: The preview processing module compares the first rotation angle and the second rotation angle; When the second rotation angle is different from the first rotation angle, the preview processing module rotates the first preview stream according to the first rotation angle to obtain the second preview stream.

4. The display device according to claim 1, characterized in that, The first shooting preview instruction also includes the application identifier of the target application; The preview processing module obtains a first rotation angle based on the first camera identifier and the current screen orientation of the display device, specifically configured as follows: Based on the application identifier of the target application, the first camera identifier, and the current screen orientation of the display device, a matching first rotation angle is queried in a preset mapping table; wherein, the preset mapping table includes rotation angles mapped by multiple shooting scene information, and the shooting scene information includes the application identifier of the application with shooting function, the camera identifier, and the screen orientation.

5. The display device according to claim 4, characterized in that, The shooting scene information also includes the functional status of the display device, which includes at least one of the on / off status of the automatic rotation function and the on / off status of the forced landscape function; when the automatic rotation function is on, the image displayed on the shooting preview interface rotates synchronously with the screen; when the forced landscape function is on, the image displayed on the shooting preview interface is rotated to the landscape orientation. The preview processing module obtains a first rotation angle based on the rear camera and the current screen orientation of the display device, specifically configured as follows: Based on the application identifier of the target application, the first camera identifier, the current screen orientation of the display device, and the current functional state of the display device, a matching first rotation angle is queried from the preset mapping table.

6. The display device according to claim 2, characterized in that, In response to receiving an operation to activate the shooting function of the target application, after determining the target camera to be used by the target application, the controller is further configured to: When the target camera is determined to be the front-facing camera, a second angle is obtained through the target application. The second angle is the angle at which the mounting angle of the gyroscope is rotated clockwise to match the imaging angle of the front-facing camera. The target application calculates the third rotation angle based on the second angle and the screen rendering angle; The target application sends a second shooting preview command to the preview processing module. The second shooting preview command includes a second camera identifier corresponding to the front camera and the third rotation angle.

7. The display device according to claim 6, characterized in that, After sending a second shooting preview command to the preview processing module via the target application, the controller is further configured to: In response to the second shooting preview command, the preview processing module sends a command to start the target camera to the driver layer. In response to the command to start the target camera, the driver layer starts the front camera and stores the first preview stream captured by the front camera into the second video node, where the second video node is the real video node associated with the front camera. The preview processing module obtains a first preview stream from the second video node, rotates the first preview stream according to the third rotation angle to obtain a third preview stream, and transmits the third preview stream to the display so that the display shows the third preview stream on the shooting preview interface.

8. The display device according to claim 1, characterized in that, The controller also operates an image processing module, and the controller is further configured to: During the process of displaying the second preview stream on the shooting preview interface, in response to receiving an image shooting operation, the target application sends a shooting instruction to the image processing module. The shooting instruction includes a first camera identifier corresponding to the rear camera and a target timestamp. The target timestamp is used to characterize the time when the target application receives the image shooting operation. In response to the photo-taking command, the photo-taking processing module obtains image frame data and thumbnail data corresponding to the target timestamp from the first video node. The image frame data is processed by the image processing module to generate an original captured image in a preset format. Based on the first rotation angle, the original captured image is rotated to obtain the target main image. The image processing module processes the thumbnail data to generate an original thumbnail corresponding to the original captured image. Based on the first rotation angle, the original thumbnail is rotated to obtain the target thumbnail.

9. The display device according to claim 7, characterized in that, The controller also operates a hardware abstraction layer, and before determining the target camera to be used by the target application in response to receiving an operation to activate the shooting function of the target application, the controller is further configured to: After the driver layer creates the first video node, it sends a video node list to the hardware abstraction layer. The video node list includes first node configuration information and second node configuration information. The first node configuration information includes the first node address and first node ID mapped by the first video node. The second node configuration information includes the second node address and second node ID mapped by the second video node. The hardware abstraction layer receives a list of video nodes, sets the rear camera attribute for the first video node, and assigns a first camera identifier to the first video node. The first camera identifier is used to indicate that the first video node is associated with the rear camera. The hardware abstraction layer sets the front-facing camera attributes for the second video node and assigns a second camera identifier to the second video node. The second camera identifier is used to indicate that the second video node is associated with the front-facing camera. The hardware abstraction layer stores a list of cameras, which includes a mapping relationship between the configuration information of the first node and the identifier of the first camera, and a mapping relationship between the configuration information of the second node and the identifier of the second camera.

10. An image processing method in a display device, characterized in that, The display device is equipped with a front-facing camera, and the method includes: The target application receives an operation to start the shooting function and determines the target camera to be used; the target camera is one of the front camera and the rear camera, and the rear camera is a virtual camera generated by the driver layer by creating a first video node, the first video node being a virtual video node associated with the rear camera; When the target application determines that the target camera is a rear camera, it sends a first shooting preview instruction to the preview processing module. The first shooting preview instruction includes a first camera identifier corresponding to the rear camera. In response to the first shooting preview command, the preview processing module sends a command to start the target camera to the driver layer; In response to the command to start the target camera, the driver layer starts the front camera and stores the first preview stream captured by the front camera into the first video node; The preview processing module obtains a first rotation angle based on the first camera identifier and the current screen orientation of the display device. The first rotation angle is a rotation angle matched by the preview processing module. The rotation angle is the angle at which the original image captured by the front camera is rotated clockwise to be consistent with the current screen orientation. The preview processing module obtains a first preview stream from the first video node, rotates the first preview stream according to the first rotation angle to obtain a second preview stream, and transmits the second preview stream to the display so that the display shows the second preview stream on the shooting preview interface.

Images