Working mode control method and device, electronic equipment and readable storage medium

Abstract

The invention relates to a working mode control method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring zoom state information and shooting scene information of a target camera, and a current working mode of a target image sensor corresponding to the target camera; under the condition that the zooming state information is in the zooming process, control request information is determined based on the zooming state information, the shooting scene information and the current working mode; and under the condition that the control state in the control request information is an interception state, determining a target working mode of the target image sensor based on the current working mode. By adopting the method, the zooming fluency can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a working mode control method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of image processing technology, traditional electronic devices are generally equipped with multiple lenses. Electronic devices can achieve zoom by selecting a fixed-focus lens or relaying a fixed-focus lens. Selecting a fixed-focus lens means selecting the most suitable fixed-focus lens to collect image data according to the zoom ratio and shooting scene during the zoom process; fixed-focus lens relay means achieving the zoom effect by switching between lenses with different focal lengths.

[0003] In traditional technology, even during the zooming process, the electronic device will continuously obtain the shooting scene information, determine the update working mode of the image sensor according to the shooting scene information, and control the image sensor to collect image data in the update working mode, resulting in freezes or lags in the picture during the zooming process, resulting in unsmooth zooming. Summary of the invention

[0004] Embodiments of the present application provide a working mode control method, device, electronic device, computer-readable storage medium, and computer program product, which can improve zoom smoothness.

[0005] In a first aspect, the present application provides a working mode control method, comprising:

[0006] Obtaining zoom status information and shooting scene information of a target camera, and a current working mode of a target image sensor corresponding to the target camera;

[0007] In a case where the zoom state information indicates that the zoom state is in the zooming process, determining control request information based on the zoom state information, the shooting scene information and the current working mode;

[0008] In a case where the control state in the control request information is an interception state, a target operating mode of the target image sensor is determined based on the current operating mode.

[0009] In a second aspect, the present application further provides a working mode control device, comprising:

[0010] An acquisition module, used to acquire zoom status information and shooting scene information of a target camera, and a current working mode of a target image sensor corresponding to the target camera;

[0011] a determination module, configured to determine control request information based on the zoom state information, the shooting scene information and the current working mode when the zoom state information indicates that the zoom state information is in the zooming process;

[0012] The control module is used to determine the target working mode of the target image sensor based on the current working mode when the control state in the control request information is an interception state.

[0013] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0015] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0016] The above-mentioned working mode control method, device, electronic device, computer-readable storage medium and computer program product, the electronic device obtains the zoom state information and shooting scene information of the target camera, and the current working mode of the target image sensor corresponding to the target camera. When the zoom state information is in the zooming process, the control request information is determined based on the zoom state information, the shooting scene information and the current working mode. When the control state in the control request information is the interception state, the target working mode of the target image sensor is determined based on the current working mode, that is, when the target camera of the electronic device is in the zooming process, the working mode of the target image sensor of the target camera is prevented from changing, so as to avoid frame loss or longer frame interval caused by working mode switching, causing the picture to be stuck or delayed during the zooming process, thereby improving the smoothness of the zoom. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] FIG1 is a schematic flow chart of a method for controlling a working mode in one embodiment;

[0019] FIG2 is a schematic diagram of a camera of an electronic device in one embodiment;

[0020] FIG3 is a flow chart of a control request information determination step in one embodiment;

[0021] FIG4 is a schematic flow chart of a target working mode determination step in one embodiment;

[0022] FIG5 is a schematic diagram of a light changing point in one embodiment;

[0023] FIG6 is a table showing the correspondence between camera types and parameters in one embodiment;

[0024] FIG7 is a schematic diagram of image data in one embodiment;

[0025] FIG8 is a schematic flow chart of a step of acquiring captured image data in one embodiment;

[0026] FIG9 is a schematic diagram of a high dynamic range image and shot count image data in one embodiment;

[0027] FIG10 is a schematic flow chart of a step of acquiring captured image data in another embodiment;

[0028] FIG11 is a schematic diagram of the structure of an electronic device in one embodiment;

[0029] FIG12 is a flow chart of the working mode control of an electronic device in one embodiment;

[0030] FIG13 is a structural block diagram of a working mode control device in one embodiment;

[0031] FIG. 14 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] In an exemplary embodiment, as shown in FIG1 , a working mode control method is provided, and the method is applied to an electronic device as an example for explanation. The electronic device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, smart cars, etc., and the portable wearable devices may be smart watches and smart bracelets, etc. It can be understood that the method can also be applied to a system including an electronic device and a server, and is implemented through the interaction between the electronic device and the server. In this embodiment, the method is applied to an electronic device as an example for explanation, including steps 102 to 106, wherein:

[0034] Step 102: Obtain zoom status information and shooting scene information of the target camera, and the current working mode of the target image sensor corresponding to the target camera.

[0035] Among them, the target camera refers to a camera in working state on the electronic device. It can be understood that there are multiple cameras on the electronic device. When the electronic device is shooting image data, at least one of the cameras is in working state, and the remaining cameras are in idle state. The camera in working state is the target camera. The target camera can also be a main camera in working state. The main camera refers to the main camera used to shoot most of the image data in the electronic device. For example, the schematic diagram of the electronic device camera is shown in Figure 2. The electronic device includes three cameras, namely, a super wide-angle lens, a wide-angle lens, and a telephoto lens. If the super wide-angle lens is used to shoot image data, the super wide-angle lens is the target camera. The zoom state information refers to the zoom operation state of the target camera. The zoom state information includes but is not limited to at least one of zoom process, zoom end, and zoom ratio. The shooting scene information refers to the data obtained by the target camera according to the external environment, scene type or scene conditions. The shooting scene information includes but is not limited to at least one of brightness information, scene information, motion information, and color information. The target image sensor refers to the image sensor of the target camera. The working mode refers to the working mode of the image sensor, including but not limited to 1dol (1 Digital Overlap), 2dol (2 Digital Overlap), QBC (Quad Bayer Coding), LBMF (Low Brightness Multi-frame), etc. The current working mode refers to the current working mode of the target image sensor.

[0036] Exemplarily, the electronic device obtains zoom status information and shooting scene information of the target camera, and a current working mode of the target image sensor corresponding to the target camera.

[0037] In an exemplary embodiment, the APS (Automatic Picture System) in the electronic device obtains the shooting scene information of the target camera from the 3A module (3A Module), and obtains the zoom state information of the target camera and the current working mode of the target image sensor corresponding to the target camera from the camera APP (APPlication). The 3A module refers to a combination module of the three automatic control functions of automatic exposure, automatic white balance and automatic focus.

[0038] In an exemplary embodiment, the APS in the electronic device obtains the shooting scene information of the target camera from the 3A module, obtains the zoom status information of the target camera from the camera APP, and obtains the current working mode of the target image sensor corresponding to the target camera from the HAL (Hardware Abstraction Layer).

[0039] In an exemplary embodiment, the user slides or scrolls the zoom control bar in the camera APP, and the electronic device zooms the target camera in response to the trigger operation of sliding or scrolling the zoom control bar, and the APS in the electronic device obtains the zoom status information of the target camera from the camera APP. Among them, the zooming method can be to select a fixed-focus lens or a fixed-focus lens relay. For example, in a SAT (Seamless Automatic Transition) hybrid zoom scenario, the user pulls up the sliding plate or a point cut occurs, and the electronic device zooms the target camera in response to the trigger operation of pulling up the sliding plate or a point cut occurs.

[0040] In an exemplary embodiment, a user clicks on a zoom area on a display screen of an electronic device, and the electronic device zooms a target camera in response to a trigger operation of clicking on the zoom area, and the APS in the electronic device obtains zoom status information of the target camera from a camera APP. The zooming method may be to select a fixed-focus lens or a fixed-focus lens relay.

[0041] Step 104: When the zoom state information indicates that the zoom process is in progress, control request information is determined based on the zoom state information, the shooting scene information, and the current working mode.

[0042] The zoom process refers to the process of changing the focal length of the target camera. The control request information refers to an instruction for changing the working mode of the target image sensor of the target camera. The control request information includes but is not limited to a change state and a control state. The change state refers to the process of changing the working mode of the image sensor, and the control state refers to information on whether to prevent the working mode of the image sensor from changing.

[0043] Exemplarily, when the electronic device determines that the zoom state information is in the zooming process, the electronic device determines to change the working mode based on the shooting scene information, and determines the control request information based on the changed working mode and the current working mode.

[0044] Step 106 : when the control state in the control request information is the interception state, determine the target working mode of the target image sensor based on the current working mode.

[0045] The interception state refers to a state in which the working mode of the target image sensor is prevented from changing, which can be understood as preventing the working mode of the target image sensor from changing, or delaying the change of the working mode of the target image sensor.

[0046] Exemplarily, the electronic device obtains the control state from the control request information, and if the control state is an interception state, determines the current working mode as the target working mode of the target image sensor, and controls the target image sensor to work in the target working mode.

[0047] In an exemplary embodiment, the camera APP in the electronic device sends control request information to HAL through the Framework layer (the framework layer of the software architecture), and HAL obtains the control status from the control request information. If the control status is an interception status, HAL determines the current working mode as the target working mode of the target image sensor, and controls the target image sensor to work in the target working mode.

[0048] In the above working mode control method, the electronic device obtains the zoom state information and shooting scene information of the target camera, and the current working mode of the target image sensor corresponding to the target camera. When the zoom state information is in the zooming process, the control request information is determined based on the zoom state information, the shooting scene information and the current working mode. When the control state in the control request information is the interception state, the target working mode of the target image sensor is determined based on the current working mode. That is, when the target camera of the electronic device is in the zooming process, the working mode of the target image sensor of the target camera is prevented from changing, so as to avoid frame loss or longer frame interval caused by working mode switching, causing the picture to be stuck or delayed during the zooming process, thereby improving the smoothness of the zoom.

[0049] In an exemplary embodiment, as shown in FIG3 , when the zoom state information indicates that the zoom state is in the zoom process, the control request information is determined based on the zoom state information, the shooting scene information and the current working mode, including:

[0050] Step 302: Determine an update working mode of the target image sensor based on the shooting scene information.

[0051] The updating working mode refers to the working mode of the target image sensor determined according to the shooting scene information.

[0052] Exemplarily, the electronic device determines an update working mode of the target image sensor according to the shooting scene information.

[0053] In an exemplary embodiment, the APS in the electronic device determines an update operation mode of the target image sensor according to the shooting scene information.

[0054] Step 304, determining a change status based on the current working mode and the updated working mode.

[0055] Among them, the change state refers to the change process of the working mode of the image sensor, and the change state includes but is not limited to the current working mode and the updated working mode. For example, if the current working mode is Binning (pixel merging) and the updated working mode is Izoom (In-sensor zoom), the change state is Binning-Izoom. For example, taking a zoom ratio of 2.5 times as an example, a 4000*3000 image data is obtained by shooting with Binning, and then after being cropped 2.5 times by the ISP (Image Signal Processor) module, the final image data with a 2.5x effect is output; while using Izoom, the camera sensor uses all its pixels to generate an 8000*6000 image data without cropping, and then crops to obtain an image with a 2x zoom ratio. The subsequent ISP module or the higher-level algorithm module performs another 1.25 times processing to obtain image data with a final effect of 2.5 times. Izoom can improve the clarity of the image, especially when the zoom ratio is large, the effect of improving clarity will be more obvious.

[0056] Exemplarily, the electronic device generates a change state according to the current working mode and the updated working mode.

[0057] Step 306, determining the control state based on the zoom state information, the current working mode and the updated working mode.

[0058] The control state refers to information on whether to prevent the working mode of the image sensor from changing. The control state can be an interception state or a non-interception state.

[0059] Exemplarily, if the zoom status information is that the zoom process is in progress, the electronic device further determines the control state based on the current working mode and the updated working mode, and the control state can be an interception state or a non-interception state; if the zoom state is the end of zoom, the electronic device directly determines the control state as a non-interception state.

[0060] Step 308, generating mode decision information based on the change status and the control status.

[0061] The mode decision information refers to information generated based on the change state and the control state.

[0062] Exemplarily, the APS in the electronic device generates mode decision information according to the change state and the control state.

[0063] Step 310: Generate control request information based on the mode decision information.

[0064] For example, the APS in the electronic device sends the mode decision information to the camera APP, and the camera APP generates control request information according to the mode decision information. That is, the APS makes a strategy and generates the mode decision information, and the camera APP makes the final check before sending it to the HAL.

[0065] In this embodiment, the electronic device determines the updated operating mode of the target image sensor according to the shooting scene information, determines the change state based on the current operating mode and the updated operating mode, and when the zoom state information is in the zoom process, determines the control state based on the current operating mode and the updated operating mode, generates mode decision information based on the change state and the control state, and generates control request information based on the mode decision information, so as to provide accurate basic information for subsequent control of the operating mode of the target image sensor.

[0066] In an exemplary embodiment, determining the control state based on the zoom state information, the current working mode and the updated working mode includes:

[0067] When the zoom state information indicates that the zoom state is in progress and the current working mode is different from the update working mode, it is determined that the control state is the interception state.

[0068] Exemplarily, the electronic device determines that the zoom state information is in the zoom process, and the current working mode and the update working mode are different, and then determines that the control state is the interception state.

[0069] In an exemplary embodiment, the electronic device determines that the zoom state information is in the zoom process, and the current working mode and the update working mode are the same, and then determines that the control state is a non-intercepting state.

[0070] In an exemplary embodiment, the APS in the electronic device determines that the zoom state information is in the zoom process, and the current working mode and the update working mode are different, and then determines that the control state is the interception state.

[0071] In this embodiment, when the zoom state information is in the zoom process, and the current working mode and the update working mode are different, it means that the target camera of the electronic device is in the zoom process, and the working mode of the target image sensor of the target camera of the electronic device is about to change from the current working mode to the update working mode to adapt to the current shooting scene. In order to prevent the working mode of the target camera from changing from the current working mode to the update working mode, the control state is determined to be the interception state, laying the foundation for the subsequent activation of the interception mechanism.

[0072] In an exemplary embodiment, generating control request information based on the mode decision information includes:

[0073] Based on the shooting scene information, the updated working mode in the mode decision information is detected to obtain a detection result; when the detection result is a mismatch, the updated working mode in the mode decision information is corrected based on the shooting scene information to obtain updated mode decision information; based on the updated mode decision information, a control request information is generated.

[0074] The detection refers to confirming whether the updated working mode matches the shooting scene information. The mismatch refers to that the updated working mode does not match the shooting scene information.

[0075] Exemplarily, after generating mode decision information, the electronic device detects the updated working mode in the mode decision information based on the shooting scene information to obtain a detection result. If the detection result is a mismatch, the updated working mode in the mode decision information is corrected based on the shooting scene information to obtain updated mode decision information, and then generates control request information based on the updated mode decision information; if the detection result is a match, the control request information is generated based on the mode decision information.

[0076] In an exemplary embodiment, after generating mode decision information, the APS in the electronic device sends the mode decision information to the camera APP. The camera APP detects the updated working mode in the mode decision information based on the shooting scene information to obtain a detection result. If the detection result is a mismatch, the updated working mode in the mode decision information is corrected based on the shooting scene information to obtain updated mode decision information, and then control request information is generated based on the updated mode decision information; if the detection result is a match, control request information is generated based on the mode decision information.

[0077] In this embodiment, the shooting scene information is used to confirm whether the updated working mode in the mode decision information matches the shooting scene information. If the updated working mode does not match the shooting scene information, it means that there is an error in the determined updated working mode. The updated working mode in the mode decision information is corrected based on the shooting scene information to obtain updated mode decision information, thereby improving the accuracy of the mode decision information. Then, control request information is generated based on the updated mode decision information, thereby improving the accuracy of the control request information.

[0078] In an exemplary embodiment, as shown in FIG4 , determining a target operating mode of a target image sensor based on a current operating mode includes:

[0079] Step 402, obtaining the zoom ratio of the target camera.

[0080] The zoom ratio refers to the magnification of the field of view by the target camera, which can be understood as the magnification of the object through the target camera. The zoom ratio can be expressed in numbers, for example, 2x means the zoom ratio is 2 times, and 5x means the zoom ratio is 5 times.

[0081] Exemplarily, the electronic device obtains the zoom ratio of the target camera.

[0082] In an exemplary embodiment, the HAL in the electronic device obtains the zoom ratio of the target camera from the camera APP.

[0083] Step 404, comparing the zoom factor with the light change point of the target camera.

[0084] Among them, the light change point refers to the zoom magnification threshold that needs to be opened for a new camera, which can be understood as the zoom magnification threshold for switching from one camera to another, or adding a zoom magnification threshold for a new camera based on the current camera. For example, the schematic diagram of the light change point is shown in Figure 5, including uw (Ultra Wide), w (Wide), T (Telephoto) and UT (Ultra Telephoto). In a stable state, the light change point corresponding to the ultra-wide-angle lens is 1.0, the light change point corresponding to the wide-angle lens is 3.0, and the light change point corresponding to the telephoto lens is 6.0, when the zoom dial is pulled up.

[0085] Exemplarily, the electronic device compares the zoom factor with the light change point of the target camera.

[0086] In an exemplary embodiment, the HAL in the electronic device compares the zoom factor with the light change point of the target camera.

[0087] Step 406: When the zoom ratio is less than the light changing point, the current working mode is determined as the target working mode of the target image sensor.

[0088] Exemplarily, the electronic device determines that the zoom ratio is less than the light changing point, then determines the current operating mode as the target operating mode of the target image sensor, and controls the target image sensor to operate in the target operating mode.

[0089] In an exemplary embodiment, the HAL in the electronic device determines that the zoom ratio is less than the light change point, then determines the current operating mode as the target operating mode of the target image sensor, and controls the target image sensor to operate in the target operating mode.

[0090] In this embodiment, when the control state in the control request information is the interception state, the electronic device compares the zoom ratio with the light change point of the target camera. If the zoom ratio is less than the light change point, it means that there is no need to turn on a new camera. The target image sensor is controlled to continue to collect image data in the current working mode, and the working mode of the target image sensor of the target camera is prevented from changing, so as to ensure the stability of the frame rate, reduce image quality fluctuations and reduce system complexity, thereby improving the smoothness of the zoom.

[0091] In an exemplary embodiment, the working mode control method further includes:

[0092] When the zoom ratio is equal to or greater than the light change point, a new camera is determined and started based on the zoom ratio; the updated working mode in the control request information is determined as the target working mode of the image sensor corresponding to the new camera; based on the camera type and zoom ratio of the target camera, the target working mode of the target image sensor is determined.

[0093] Here, starting a new camera refers to changing an idle camera to a working state. The camera type refers to the type of the camera, and the camera type includes but is not limited to one of an ultra-wide-angle lens, a wide-angle lens, a T telephoto lens, and an ultra-telephoto lens.

[0094] Exemplarily, the electronic device compares the zoom ratio with the light change point of the target camera. If the zoom ratio is equal to or greater than the light change point, a new camera is determined and started based on the zoom ratio, and the update working mode in the control request information is determined as the target working mode of the image sensor corresponding to the new camera, and the new camera is controlled to collect image data in the update working mode; the camera type of the target camera is obtained, and the target working mode of the target image sensor is determined based on the camera type and zoom ratio of the target camera.

[0095] In this embodiment, the electronic device compares the zoom ratio with the light change point of the target camera. If the zoom ratio is equal to or greater than the light change point, it means that a new camera needs to be turned on to meet the zoom ratio requirement. In order to match the working mode of the new camera with the shooting scene information, the image sensor corresponding to the new camera is controlled to collect image data in an updated working mode. That is, for the streaming scene during the zoom process, the image sensor corresponding to the new camera is changed from an off state to an on state, and the streaming can be started based on a specific updated working mode. The image sensor corresponding to the new camera is configured based on the current shooting scene, thereby improving the accuracy of image data collection. For the target sensor, the target working mode of the target image sensor is determined based on the camera type and zoom ratio of the target camera.

[0096] In an exemplary embodiment, determining a target operating mode of a target image sensor based on a camera type and a zoom ratio of a target camera includes:

[0097] When the camera type of the target camera is a telephoto lens and the update working mode is in-sensor zoom, the zoom ratio and the ratio threshold are compared; when the zoom ratio is greater than the ratio threshold, the update working mode in the control request information is determined as the target working mode of the target image sensor; when the zoom ratio is less than or equal to the ratio threshold, the current working mode is determined as the target working mode of the target image sensor.

[0098] The magnification threshold refers to the minimum zoom magnification of the image sensor of the telephoto lens, which is immediately switched to the in-sensor zoom. The magnification threshold can be set according to the clarity requirements of the image data. In-sensor zoom (Izoom) refers to the working mode in which the image sensor cuts out a smaller area from all the image data collected by the image sensor to achieve a magnification effect. For example, as shown in the corresponding relationship table of camera types and parameters in FIG6 , when the working mode of the image sensor of the ultra-wide-angle lens is 16M normal (16 Megapixel Normal Mode, 16 million pixel standard mode), the zoom factor is less than 1; when the working mode of the image sensor of the wide-angle lens is 12M-binning mode (12 Megapixel Pixel Binning Mode, 12 million pixel pixel merging mode), the zoom factor is 1-1.99; when the working mode of the image sensor of the wide-angle lens is sensor crop to 12M from 48M (Sensor Cropping to 12 Megapixel from 48 Megapixel, sensor cropped from 48 million pixels to 12 million pixels), the zoom factor is 2-2.99; when the working mode of the image sensor of the telephoto lens is 12M normal (12Megapixel Normal Mode, 12 million pixel standard mode), the zoom factor is 3-10. If the zoom ratio is 1.5, the zoom ratio requirement can be met by turning on the wide-angle lens. For the telephoto lens, when the zoom ratio is greater than the ratio threshold, the zoom ratio requirement can no longer be met by turning on a new camera. If, when the zoom ratio is greater than the ratio threshold, the image sensor of the telephoto lens is still controlled to use the image data collected in the current working mode, the clarity of the collected image data will be reduced. In order to improve the clarity of the collected image data, when the zoom ratio is greater than the ratio threshold, the working mode of the image sensor of the telephoto lens needs to be immediately switched to in-sensor zoom.

[0099] Exemplarily, when the camera type of the target camera is a telephoto lens and the update working mode is in-sensor zoom, the magnification threshold is obtained, the zoom magnification is compared with the magnification threshold, and if the zoom magnification is greater than the magnification threshold, the update working mode in the control request information is determined as the target working mode of the target image sensor, and the target image sensor is controlled to collect image data in the update working mode; if the zoom magnification is less than or equal to the magnification threshold, the current working mode is determined as the target working mode of the target image sensor, and the target image sensor is controlled to collect image data in the current working mode. For example, as shown in the schematic diagram of image data in FIG7, the left figure is image data collected by the telephoto lens using in-sensor zoom, and the right figure is image data collected by the telephoto lens using Binning, and the clarity of the image data in the left figure is higher than that of the image data in the right figure.

[0100] In this embodiment, when the camera type of the target camera is a telephoto lens and the updated working mode is in-sensor zoom, if the zoom ratio is greater than the ratio threshold, the target image sensor is still controlled to use the image data collected by the current working mode, and the clarity of the collected image data will be reduced. Therefore, when the zoom ratio is greater than the ratio threshold, the working mode of the target image sensor is immediately switched to in-sensor zoom, thereby improving the clarity of the collected image data; if the zoom ratio is less than or equal to the ratio threshold, the target image sensor is controlled to use the image data collected by the current working mode, avoiding the switching of the working mode of the target image sensor, thereby improving the smoothness of the zoom.

[0101] In an exemplary embodiment, after determining the target operating mode of the target image sensor based on the current operating mode, the method further includes:

[0102] The update zoom state information is obtained. When the update zoom state information indicates that the zoom is finished and the working mode of the target sensor is the current working mode, the update working mode in the control request information is determined as the target working mode of the target image sensor.

[0103] The updated zoom state information refers to the information obtained again and representing the zoom state. The updated zoom state information may be an updated focal length or a zoom state. The zoom state may be in the zoom process or at the end of the zoom.

[0104] Exemplarily, the electronic device obtains updated zoom status information. If the updated zoom status information indicates that zoom is ended, the operating mode of the target sensor is obtained. If the operating mode of the target sensor is the current operating mode, the updated operating mode in the control request information is determined as the target operating mode of the target image sensor, and the target image sensor is controlled to collect image data in the updated operating mode.

[0105] In this embodiment, if the zoom of the electronic device is completed and the operating mode of the target sensor is the current operating mode, the update operating mode in the control request information is determined as the target operating mode of the target image sensor, and the target image sensor is controlled to collect image data in the update operating mode. The target operating mode matches the shooting scene information, thereby improving the accuracy of the collected image data.

[0106] In an exemplary embodiment, obtaining and updating zoom state information includes:

[0107] Gets updated zoom status information from the hardware abstraction layer.

[0108] Among them, the Hardware Abstractio Layer (HAL) refers to the software layer located between the camera APP and the image sensor, which realizes the interaction between the camera APP and the image sensor through a unified interface. It can be understood that the camera APP sends the control request information to the hardware abstraction layer, and the hardware abstraction layer controls the image sensor according to the control request information.

[0109] Exemplarily, the electronic device obtains updated zoom status information from the hardware abstraction layer.

[0110] In an exemplary embodiment, the APS of the electronic device obtains updated zoom state information from the hardware abstraction layer.

[0111] In this embodiment, the hardware abstraction layer controls the camera zoom, and the hardware abstraction layer knows the zoom status information first. Compared with obtaining the zoom status information from the camera APP, it is necessary to wait for the hardware abstraction layer to feed back the zoom status information to the camera APP, and then obtain the zoom status information from the camera APP. Obtaining the zoom status information from the hardware abstraction layer improves the efficiency of obtaining the zoom status information, and can switch the working mode of the target image sensor to the update working mode more quickly.

[0112] In an exemplary embodiment, as shown in FIG8 , the working mode control method further includes:

[0113] Step 802: When the zoom status information indicates that the zoom state is in progress, a shooting event is obtained.

[0114] The shooting event refers to an event of collecting image data, and the image data collected by the shooting event may be live photo data.

[0115] Exemplarily, when the zoom state information indicates that the electronic device is in the zooming process, the electronic device obtains a shooting event in response to a triggering operation of a shooting control.

[0116] Step 804: when the control state in the control request information is the interception state, determine the field of view angle of the image to be captured of the image data to be captured based on the update working mode in the control request information.

[0117] The image data to be captured refers to the image data collected by the target image sensor in the update working mode. The field of view of the image to be captured refers to the range covered by the image sensor when collecting the image data to be captured. The field of view of the image to be captured can be expressed in degrees. The larger the field of view of the image to be captured, the wider the image range collected by the target image sensor. The smaller the field of view of the image to be captured, the smaller the image range collected by the target image sensor.

[0118] Exemplarily, when the control state in the control request information is the interception state, the electronic device determines the field of view angle of the image to be captured of the image data to be captured collected by the target image sensor in the update working mode.

[0119] Step 806: Determine the current image field of view of the currently captured image data based on the current working mode.

[0120] The current captured image data refers to the image data collected by the target image sensor in the current working mode. The current image field of view refers to the range covered by the image sensor when collecting the current image data, and the current image field of view can be expressed in degrees.

[0121] Exemplarily, the electronic device determines a current image field angle of current captured image data acquired by the target image sensor in a current working mode.

[0122] Step 808 : When the field of view angle of the image to be captured is different from the field of view angle of the current image, based on the update working mode, the captured image data corresponding to the capture event is acquired.

[0123] Exemplarily, the electronic device compares the field of view of the image to be captured with the field of view of the current image. If the field of view of the image to be captured is different from the field of view of the current image, then based on the updated working mode, the captured image data corresponding to the shooting event is obtained; if the field of view of the image to be captured is the same as the field of view of the current image, then based on the current working mode, the captured image data corresponding to the shooting event is obtained.

[0124] In an exemplary embodiment, after step 808, it also includes: performing high dynamic range imaging on the captured image data corresponding to the captured event to obtain a high dynamic range image corresponding to the captured event. High dynamic range (HDR) imaging is a technology used to achieve a larger exposure dynamic range than ordinary digital image technology, and is suitable for use in scenes with obvious changes in shade, so that the bright scene will not be overexposed, and the dark scene will not be underexposed, or the details of the scene in the dark can be more fully expressed. For example, when shooting a person in a backlit environment, both the person and the environment can be clearly photographed. High dynamic range images can provide more dynamic range and image details than captured image data. For example, a schematic diagram of a high dynamic range image and shot image data is shown in Figure 9, where the left figure in Figure 9 is the captured image data and the right figure is the high dynamic range image. The high dynamic range image presents more image details than the captured image.

[0125] In an exemplary embodiment, based on the update working mode, captured image data corresponding to a shooting event is obtained, including: the electronic device obtains the image data collected by the target image sensor in the update working mode by delaying the photo response, and determines the image data collected by the target image sensor in the update working mode as the captured image data corresponding to the shooting event.

[0126] In an exemplary embodiment, based on the update working mode, captured image data corresponding to the shooting event is obtained, including: the electronic device obtains the image data collected by the target image sensor in the update working mode through a post-frame method, and determines the image data collected by the target image sensor in the update working mode as the captured image data corresponding to the shooting event.

[0127] In an exemplary embodiment, a flow chart of the steps of acquiring captured image data is shown in FIG10 , including: when the zoom state information is in the zoom process, acquiring a capture event; when the control state in the control request information is an interception state, taking a normal response to the capture, and acquiring the captured image data corresponding to the capture event by means of a front frame fetching method; when the control state in the control request information is an interception state, determining whether the update working mode is an in-sensor zoom; if the update working mode is not an in-sensor zoom, acquiring the image data collected by the target image sensor in the update working mode by means of a delayed response to the capture and a post-frame fetching method, and determining the captured image data corresponding to the capture event by means of the image data collected by the target image sensor in the update working mode; if the update working mode is an in-sensor zoom, using a normal response to the capture, acquiring the image data collected by the target image sensor in the update working mode by means of a front frame fetching method, and determining the captured image data corresponding to the capture event by means of the image data collected by the target image sensor in the update working mode. That is, the working mode is changed to in-sensor zoom, and the working mode of the target image sensor is directly switched to the update working mode. The captured image data corresponding to the shooting event can be obtained by using the front frame method. If the working mode is changed to non-in-sensor zoom, it is necessary to wait until the zoom is completed and then switch the working mode of the target image sensor to the update working mode. Therefore, it is necessary to delay the response by taking a photo and obtain the captured image data corresponding to the shooting event by using the post-frame method.

[0128] In this embodiment, the electronic device obtains a shooting event when the zoom state information is in the zoom process, determines the field of view of the image to be shot of the image data to be shot collected by the target image sensor in the update working mode, and the current image field of view of the current shot image data collected by the target image sensor in the current working mode. If the field of view of the image to be shot is different from the field of view of the current image, it means that the field of view of the image data collected using the update working mode is different from the field of view of the image data collected using the current working mode. In order to ensure that the field of view of the image data for subsequent image fusion is the same and to avoid ghosting of the image data obtained by image fusion, the image data collected using the update working mode needs to be used as the shot image data corresponding to the shooting event, thereby improving the accuracy of the image data.

[0129] In an exemplary embodiment, the structural diagram of the electronic device is shown in FIG11 , including: APS, 3A module, camera APP, Framework layer, HAL, driver and multiple image sensors, and the driver is used to drive the multiple image sensors.

[0130] The working mode control flow chart of the electronic device is shown in FIG12 , including: the electronic device responds to the triggering operation of the zoom control, the electronic device zooms, during the zooming process of the electronic device, the APS in the electronic device obtains the shooting scene information of the target camera from the 3A module, obtains the zoom state information of the target camera and the current working mode of the target image sensor corresponding to the target camera from the camera APP, and when the APS determines that the zoom state information is in the zooming process, the APS determines the update working mode of the target image sensor according to the shooting scene information, and generates a change state according to the current working mode and the update working mode; if the current working mode and the update working mode are different, the control state is determined to be an interception state; if the current working mode and the update working mode are the same, the control state is determined to be a non-interception state; the APS generates mode decision information according to the change state and the control state. The APS sends the mode decision information to the camera APP.

[0131] The camera APP in the electronic device detects the updated working mode in the mode decision information based on the shooting scene information to obtain the detection result. If the detection result is not matched, the updated working mode in the mode decision information is corrected based on the shooting scene information to obtain the updated mode decision information, and then the control request information is generated based on the updated mode decision information; if the detection result is matched, the control request information is generated based on the mode decision information. The camera APP sends the control request information to the HAL through the Framework layer (the framework layer of the software architecture).

[0132] The HAL in the camera APP in the electronic device obtains the control status from the control request information. If the control status is the interception status, the HAL obtains the zoom ratio of the target camera from the camera APP, and compares the zoom ratio with the light change point of the target camera. If the zoom ratio is less than the light change point, the current working mode is determined as the target working mode of the target image sensor, and the target image sensor is controlled to work in the target working mode.

[0133] If the zoom ratio is equal to or greater than the light change point, a new camera is determined and started based on the zoom ratio, and the update working mode in the control request information is determined as the target working mode of the image sensor corresponding to the new camera, and the new camera is controlled to collect image data in the update working mode. The camera type of the target camera is obtained. If the camera type of the target camera is other lenses except the telephoto lens, the current working mode is determined as the target working mode of the target image sensor corresponding to the target camera, and the target camera is controlled to collect image data in the current working mode; if the camera type of the target camera is a telephoto lens and the update working mode is in-sensor zoom, the magnification threshold is obtained, and the zoom ratio is compared with the magnification threshold. If the zoom ratio is greater than the magnification threshold, the update working mode in the control request information is determined as the target working mode of the target image sensor, and the target image sensor is controlled to collect image data in the update working mode; if the zoom ratio is less than or equal to the magnification threshold, the current working mode is determined as the target working mode of the target image sensor, and the target image sensor is controlled to collect image data in the current working mode.

[0134] In the above working mode control method, the electronic device obtains the zoom state information and shooting scene information of the target camera, and the current working mode of the target image sensor corresponding to the target camera. When the zoom state information is in the zooming process, the control request information is determined based on the zoom state information, the shooting scene information and the current working mode. When the control state in the control request information is the interception state, the target working mode of the target image sensor is determined based on the current working mode. That is, when the target camera of the electronic device is in the zooming process, the working mode of the target image sensor of the target camera is prevented from changing, so as to avoid frame loss or longer frame interval caused by working mode switching, causing the picture to be stuck or delayed during the zooming process, thereby improving the smoothness of the zoom.

[0135] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0136] Based on the same inventive concept, the embodiment of the present application also provides a working mode control device for implementing the working mode control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more working mode control device embodiments provided below can refer to the limitations of the working mode control method above, and will not be repeated here.

[0137] In an exemplary embodiment, as shown in FIG13 , a working mode control device is provided, comprising: an acquisition module, a determination module and a control module, wherein:

[0138] An acquisition module 1302 is used to acquire zoom state information and shooting scene information of a target camera, and a current working mode of a target image sensor corresponding to the target camera;

[0139] A determination module 1304 is used to determine control request information based on the zoom state information, the shooting scene information and the current working mode when the zoom state information indicates that the zoom state is in the zoom process;

[0140] The control module 1306 is configured to determine a target operating mode of the target image sensor based on the current operating mode when the control state in the control request information is an interception state.

[0141] In an exemplary embodiment, the determination module 1304 is also used to: determine the update working mode of the target image sensor based on the shooting scene information; determine the change state based on the current working mode and the update working mode; determine the control state based on the zoom state information, the current working mode and the update working mode; generate mode decision information based on the change state and the control state; and generate control request information based on the mode decision information.

[0142] In an exemplary embodiment, the determination module 1304 is further used to: when the zoom state information is in the zoom process, and the current working mode and the update working mode are different, determine that the control state is the interception state.

[0143] In an exemplary embodiment, the determination module 1304 is also used to: based on the shooting scene information, detect the updated working mode in the mode decision information to obtain a detection result; when the detection result is a mismatch, correct the updated working mode in the mode decision information based on the shooting scene information to obtain updated mode decision information; based on the updated mode decision information, generate control request information.

[0144] In an exemplary embodiment, the control module 1306 is also used to: obtain the zoom ratio of the target camera; compare the zoom ratio with the light change point of the target camera; and when the zoom ratio is less than the light change point, determine the current operating mode as the target operating mode of the target image sensor.

[0145] In an exemplary embodiment, the control module 1306 is also used to: when the zoom ratio is equal to or greater than the light change point, determine and start a new camera based on the zoom ratio; determine the updated working mode in the control request information as the target working mode of the image sensor corresponding to the new camera; and determine the target working mode of the target image sensor based on the camera type and zoom ratio of the target camera.

[0146] In an exemplary embodiment, the control module 1306 is also used to: when the camera type of the target camera is a telephoto lens and the update operating mode is in-sensor zoom, compare the zoom ratio and the magnification threshold; when the zoom ratio is greater than the magnification threshold, determine the update operating mode in the control request information as the target operating mode of the target image sensor; when the zoom ratio is less than or equal to the magnification threshold, determine the current operating mode as the target operating mode of the target image sensor.

[0147] In an exemplary embodiment, the control module 1306 is also used to: obtain updated zoom status information, and when the updated zoom status information is the end of zoom and the working mode of the target sensor is the current working mode, determine the updated working mode in the control request information as the target working mode of the target image sensor.

[0148] In an exemplary embodiment, the control module 1306 is further configured to obtain update zoom status information from the hardware abstraction layer.

[0149] In an exemplary embodiment, the working mode control device also includes a shooting module, which is used to: obtain a shooting event when the zoom state information is in the zoom process; determine the field of view angle of the image to be shot of the image data to be shot based on the updated working mode in the control request information when the control state in the control request information is the interception state; determine the current image field of view angle of the current shot image data based on the current working mode; and obtain the shot image data corresponding to the shooting event based on the updated working mode when the field of view angle of the image to be shot is different from the field of view angle of the current image.

[0150] Each module in the above-mentioned working mode control device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0151] In an exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG14. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a working mode control method is implemented. The display unit of the electronic device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0152] Those skilled in the art will understand that the structure shown in FIG. 14 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0154] In one embodiment, an electronic device is provided, including a processor, a display driver chip and a display screen, characterized in that the processor executes the steps in the above-mentioned method embodiments when executing a computer program; the display driver chip is used to control a target area in the display screen to refresh according to a target refresh frequency, and the target area and the target refresh frequency are determined by the steps in the above-mentioned method embodiments.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0156] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0158] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0160] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A working mode control method, characterized in that: include: Obtaining zoom status information and shooting scene information of a target camera, and a current working mode of a target image sensor corresponding to the target camera; In a case where the zoom state information indicates that the zoom state is in the zooming process, determining control request information based on the zoom state information, the shooting scene information and the current working mode; In a case where the control state in the control request information is an interception state, a target operating mode of the target image sensor is determined based on the current operating mode.

2. The method according to claim 1, characterized in that In the case where the zoom state information is in the zooming process, determining the control request information based on the zoom state information, the shooting scene information and the current working mode includes: Based on the shooting scene information, determining an update working mode of the target image sensor; Determining a change state based on the current working mode and the updated working mode; Determining a control state based on the zoom state information, the current working mode and the updated working mode; generating mode decision information based on the change state and the control state; Based on the mode decision information, control request information is generated.

3. The method according to claim 2, characterized in that The determining of the control state based on the zoom state information, the current working mode and the updated working mode includes: When the zoom state information indicates that the zoom state is in progress and the current working mode is different from the updated working mode, it is determined that the control state is an interception state.

4. The method according to claim 2, characterized in that: The generating control request information based on the mode decision information includes: Based on the shooting scene information, detecting the updated working mode in the mode decision information to obtain a detection result; In the case where the detection result is mismatch, modifying the updated working mode in the mode decision information based on the shooting scene information to obtain updated mode decision information; Based on the updated mode decision information, control request information is generated.

5. The method according to claim 1, characterized in that The step of determining a target operating mode of the target image sensor based on the current operating mode includes: Obtaining the zoom ratio of the target camera; Comparing the zoom factor with the light change point of the target camera; When the zoom ratio is smaller than the light changing point, the current operating mode is determined as the target operating mode of the target image sensor.

6. The method according to claim 5, characterized in that The method further comprises: When the zoom ratio is equal to or greater than the light change point, determining and starting a new camera based on the zoom ratio; Determine the updated working mode in the control request information as the target working mode of the image sensor corresponding to the new camera; Based on the camera type of the target camera and the zoom ratio, a target operating mode of the target image sensor is determined.

7. The method according to claim 6, characterized in that The step of determining a target operating mode of the target image sensor based on the camera type and the zoom ratio of the target camera includes: When the camera type of the target camera is a telephoto lens and the update operation mode is in-sensor zoom, comparing the zoom magnification with a magnification threshold; When the zoom magnification is greater than the magnification threshold, determining the updated operating mode in the control request information as the target operating mode of the target image sensor; When the zoom magnification is less than or equal to the magnification threshold, the current operating mode is determined as the target operating mode of the target image sensor.

8. The method according to claim 1, characterized in that After determining the target operating mode of the target image sensor based on the current operating mode, the method further includes: Acquire updated zoom status information, and when the updated zoom status information indicates zoom end and the working mode of the target sensor is the current working mode, determine the updated working mode in the control request information as the target working mode of the target image sensor.

9. The method according to claim 8, characterized in that The obtaining and updating of zoom status information includes: Gets updated zoom status information from the hardware abstraction layer.

10. The method according to claim 1, characterized in that The method further comprises: When the zoom state information indicates that the zoom state is in the process of being zoomed, obtaining a shooting event; In a case where the control state in the control request information is an interception state, determining a field of view angle of the image to be captured based on the update working mode in the control request information; Based on the current working mode, determining a current image field of view angle of the currently captured image data; In a case where the field of view angle of the image to be captured is different from the field of view angle of the current image, based on the update working mode, captured image data corresponding to the capturing event is acquired.

11. A working mode control device, characterized in that: The device comprises: An acquisition module, used to acquire zoom status information and shooting scene information of a target camera, and a current working mode of a target image sensor corresponding to the target camera; a determination module, configured to determine control request information based on the zoom state information, the shooting scene information and the current working mode when the zoom state information indicates that the zoom state information is in the zooming process; The control module is used to determine the target working mode of the target image sensor based on the current working mode when the control state in the control request information is an interception state.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images