Camera apparatus and control method therefor, and electronic device

Abstract

Provided in the present application are a camera apparatus and a control method therefor, and an electronic device, which can improve the precision of optical image stabilization of a camera apparatus and the imaging sharpness of the camera apparatus. The method is applied to an electronic device, wherein the electronic device comprises a camera apparatus, the camera apparatus comprises an imaging sensor and an image stabilization element, and the imaging sensor comprises a dynamic vision sensor (DVS). The method comprises: determining a target motion vector; determining a feedback motion vector in a captured image by means of a DVS; on the basis of the feedback motion vector, driving an image stabilization element to move; and when the feedback motion vector is the same as the target motion vector, outputting an image. The electronic device acquires a feedback motion vector in a captured image within an extremely short time, and drives in real time an image stabilization element to move, thereby improving the sharpness during photograph capturing.

Description

Camera devices and their control methods, electronic devices

[0001] This application claims priority to Chinese Patent Application No. 202410815380.8, filed on June 21, 2024, entitled "Camera Device and Control Method Thereof, Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic devices, and more specifically, to a camera device and its control method, and electronic devices. Background Technology

[0003] As cameras are increasingly used in electronic devices, the demands on image quality are also rising. Optical image stabilization (OIS) is an optical image stabilization technology that uses a movable lens assembly mounted on the camera lens or image sensor to counteract camera shake during handheld shooting, thereby reducing image blur and out of focus. However, current OIS technology faces the following challenges: due to hardware limitations and noise, it struggles to handle real-world camera shake, resulting in continued blurring even with longer exposure times; and because traditional cameras have low frame rates, they struggle to track fast-moving objects and acquire motion vectors quickly enough, failing to achieve sharp images when the subject is moving rapidly. Achieving sharp imaging with cameras remains a pressing issue. Summary of the Invention

[0004] This application provides a camera device and control method, as well as an electronic device, which can improve the accuracy of optical image stabilization and the image clarity of the camera device.

[0005] In a first aspect, a control method for a camera device is provided, the camera device including an imaging sensor and an image stabilization element, the imaging sensor including a dynamic visual sensor (DVS), the method including: determining a target motion vector; determining a feedback motion vector in the captured image using the DVS; driving the image stabilization element to move according to the feedback motion vector; and outputting an image when the feedback motion vector is the same as the target motion vector.

[0006] Based on the above technical solution, since the DVS sensor has the characteristics of high dynamic range and low latency, it enables the camera device to acquire the feedback motion vector in the subject image in a very short time and drive the image stabilization element to move in real time, thereby improving the clarity of the captured photos.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, driving the image stabilization element to move according to the feedback motion vector includes: controlling the movement of the image stabilization element according to the opposite direction of the projection direction of the feedback motion vector.

[0008] Based on the above technical solution, the electronic device can more accurately control the movement of the image stabilization element according to the opposite direction of the feedback motion vector projection direction, thereby improving the image stabilization effect and improving the clarity of the image.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, driving the image stabilization element to move according to the feedback motion vector includes: driving the image stabilization element to move according to the difference between the feedback motion vector and the target motion vector.

[0010] Based on the above technical solution, electronic devices can more accurately control the movement of the image stabilization element according to the difference between the feedback motion vector and the target motion vector, thereby improving the image stabilization effect and the clarity of the image.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, determining the feedback motion vector in the captured image includes: determining the feedback motion vector based on event information, wherein the event information includes event accumulation frames or event-by-event streams.

[0012] Based on the above technical solution, the electronic device determines the feedback motion vector in the captured image according to the event information output by the DVS sensor, thereby enabling the electronic device to further determine the amount of motion of the image stabilization element and achieve the image stabilization effect.

[0013] In some possible implementations, determining the feedback motion vector in the captured image also includes: determining the feedback motion vector based on image information and event information.

[0014] Based on the above technical solutions, electronic devices can further combine richer details from image information on top of event information, making the calculated feedback motion vector more accurate, thereby improving the image stabilization effect.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, determining the feedback motion vector in the captured image includes: determining the feedback motion vector based on the event information when the camera device is in the exposure period.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the imaging sensor further includes an image sensor (CIS) for determining image information in the captured image. When the imaging device captures an image before exposure, determining the feedback motion vector in the captured image includes: determining the feedback motion vector based on a preview stream image and event information, wherein the event information includes event accumulation frames or event-by-event streams.

[0017] Based on the above technical solution, before exposure, the camera continuously outputs a preview stream image. The electronic device can further combine richer details in the preview stream image based on event information, making the calculated feedback motion vector more accurate, thereby improving the image stabilization effect.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, driving the image stabilization element to move according to the feedback motion vector further includes: determining the source of the feedback motion vector; the feedback motion vector includes a foreground feedback motion vector or a background feedback motion vector; and driving the image stabilization element to move according to the source of the feedback motion vector.

[0019] Based on the above technical solution, the electronic device can distinguish feedback motion vectors from different sources in the captured image, enabling the electronic device to achieve image stabilization in different scenarios according to the source of different feedback motion vectors.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, driving the image stabilization element to move according to the source of the feedback motion vector includes: driving the image stabilization element to move when the feedback motion vector is the foreground feedback motion vector, and saving the background feedback motion vector.

[0021] Based on the above technical solution, the electronic device determines that image stabilization is needed for the foreground object. However, in subsequent processing, the user may also need to stabilize the background object. Therefore, it is necessary to save the background feedback motion vector in the captured image so that the background feedback motion vector can be further processed in the future, thereby achieving the effect of panoramic image stabilization in the captured image.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: performing anti-shake processing on the background feedback motion vector.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, driving the image stabilization element to move according to the source of the feedback motion vector includes: when the feedback motion vector is the background feedback motion vector and the image stabilization element is driven to move, saving the foreground feedback motion vector.

[0024] Based on the above technical solution, the electronic device determines that it needs to perform image stabilization on the background object. However, in the subsequent processing, the user may also need to perform image stabilization on the foreground object. Therefore, it is also necessary to save the foreground feedback motion vector in the captured image so that the foreground feedback motion vector can be further processed in the subsequent process, thereby achieving the effect of panoramic image stabilization in the captured image.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: performing anti-shake processing on the foreground feedback motion vector.

[0026] In a second aspect, a camera device is provided, the camera device including a first camera module, the first camera module including a first lens group, a first imaging sensor and a first image stabilization element, the first imaging sensor including a first dynamic visual sensor (DVS), the first lens group being used to collect incident light onto the photosensitive unit of the first imaging sensor; the first dynamic visual sensor (DVS) being used to determine a feedback motion vector in the captured image; the first image stabilization element being used to drive the components in the first camera module to move according to the feedback motion vector.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the first image stabilization element is further used to drive the components in the first camera module to move in the opposite direction of the projection direction of the feedback motion vector.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the first image stabilization element is further configured to drive the components in the first camera module to move based on the difference between the feedback motion vector and the target motion vector.

[0029] Based on the above technical solution, the DVS sensor in the camera device can acquire the feedback motion vector in the captured image in a very short time and drive the first image stabilization element to move in real time, thereby improving the clarity of the captured photos.

[0030] In conjunction with the second aspect, in some implementations of the second aspect, the camera device further includes a second image stabilization element, which is used to drive the camera module in the camera device to move according to the feedback motion vector.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the second image stabilization element includes a gimbal optical image stabilization motor.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, the second image stabilization element is also used to drive the camera module in the camera device to move in the opposite direction of the projection direction of the feedback motion vector.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the second image stabilization element is further used to drive the camera module in the camera device to move based on the difference between the feedback motion vector and the target motion vector.

[0034] Based on the above technical solution, the second image stabilization element can drive the overall movement of the camera module. When the camera device includes multiple camera modules, the second image stabilization element can make the multiple camera modules move with the same amount of motion, thereby improving the consistency of the image stabilization effect.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, the first imaging sensor further includes a first image sensor (CIS) for acquiring image information in the captured image.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the first image stabilization element includes at least one of a sensor optical image stabilization motor, a lens optical image stabilization motor, or a prism optical image stabilization motor.

[0037] Based on the above technical solution, the first image stabilization element can drive the sensor, lens or prism so that these components can move according to the motion vector, so that the camera device can achieve image stabilization effect.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the camera device further includes a second camera module, the second camera module including a second lens group, a second imaging sensor and the first image stabilization element, the second imaging sensor including at least a second image sensor (CIS); the second lens group is used to collect incident light onto the photosensitive unit of the second imaging sensor; the second image sensor (CIS) is used to acquire image information in the captured image; the first image stabilization element of the second camera module is used to drive the components in the second camera module to move according to the feedback motion vector.

[0039] In conjunction with the second aspect, in some implementations of the second aspect, the first image stabilization element of the second camera module is further used to drive the components in the second camera module to move in the opposite direction of the projection direction of the feedback motion vector.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the first image stabilization element of the second camera module is further used to drive the components in the second camera module to move according to the difference between the feedback motion vector and the target motion vector.

[0041] Based on the above technical solution, the CIS sensor and the DVS sensor can be set in two different camera modules to acquire image data information and event information respectively, thereby improving the flexibility of camera device design.

[0042] In conjunction with the second aspect, in some implementations of the second aspect, the second imaging sensor further includes a second dynamic vision sensor (DVS) for determining feedback motion vectors in the captured image.

[0043] Based on the above technical solution, the camera device can simultaneously output event information and image information, and determine the feedback motion vector in the captured image based on the event information and image information.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the dynamic vision sensor (DVS) and the image sensor (CIS) are fused together.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the dynamic vision sensor (DVS) is set separately from the image sensor (CIS).

[0046] Based on the above technical solution, the DVS sensor and CIS sensor can be set in the imaging sensor of the camera device and work simultaneously, so that the electronic device can determine the feedback motion vector in the captured image based on the image information and event information output by the imaging sensor, so that the electronic device can perform image stabilization.

[0047] Thirdly, an electronic device is provided, the electronic device comprising a camera device as described in the second aspect and any implementation thereof, one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs comprising instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in the first aspect and any implementation thereof.

[0048] Fourthly, a computer-readable storage medium is provided, wherein a program or instructions are stored therein, which, when executed, implement the method as described in the first aspect and any of its implementations.

[0049] Fifthly, a chip is provided, wherein the chip stores instructions that, when executed, implement the method described in the first aspect and any of its implementations.

[0050] In a sixth aspect, a computer program product is provided, wherein the computer program product stores a program or instructions that, when the program or instructions are run, implement the method as described in the first aspect and any of its implementations. Attached Figure Description

[0051] Figure 1 is a schematic diagram of an electronic device provided in an embodiment of this application.

[0052] Figure 2 is a schematic diagram of a camera device provided in an embodiment of this application.

[0053] Figure 3 is a schematic diagram of an imaging sensor provided in an embodiment of this application.

[0054] Figure 4 is a flowchart illustrating a control method for a camera device provided in an embodiment of this application.

[0055] Figure 5 is a schematic diagram of the working principle of the DVS sensor.

[0056] Figure 6 is a schematic block diagram of another electronic device provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0058] The embodiments of this application are described in detail below, and examples of these embodiments are illustrated in the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0059] In the description of this application, it should be understood that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0060] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between the components; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0061] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. In the description of this application, it should be understood that the terms “center,” “longitudinal,” “lateral,” “upper,” “lower,” “front,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0062] Figure 1 is a schematic diagram of an electronic device provided in an embodiment of this application. This embodiment uses a mobile phone as an example for description.

[0063] The electronic device 100 includes a housing 10, a display screen 20, an image processor 30, and a camera device 40. In some embodiments, the housing 10 includes a frame 101 and a back cover 102. The frame 101 and the back cover 102 can be integrally formed or assembled into an integral structure. The display screen 20 and the back cover 102 are respectively mounted on both sides of the frame 101, together enclosing the internal cavity of the device. The display screen 20 can be a liquid crystal display (LCD) screen, an organic light emitting diode (OLED) screen, etc., wherein the OLED screen can be a flexible display screen or a rigid display screen.

[0064] The image processor 30 and the camera device 40 are housed within the overall cavity of the device. The image processor 30 is communicatively connected to the camera device 40, and is used to acquire and process image data from the camera device 40. The communication connection between the camera device 40 and the image processor 30 can include data transmission via electrical connections such as wiring, or data transmission via coupling. It is understood that the camera device 40 and the image processor 30 can also be connected via other methods capable of data transmission.

[0065] The image processor 30 optimizes the digital image signal and transmits the processed signal to the display. The image processor 30 can be an image processing chip or a digital signal processing chip. Its function is to transmit the data obtained by the image sensor to the central processing unit in a timely and fast manner and refresh the image sensor. Therefore, the quality of the image processor 30 chip directly affects the image quality (such as color saturation, sharpness, etc.).

[0066] The camera device 40 can be installed only on the front of the electronic device 100 to capture the scene located on one side of the front of the electronic device 100, and in some embodiments it can be called a front camera device; it can also be installed only on the back of the electronic device 100 to capture the scene located on one side of the back of the electronic device 100, and in some embodiments it can be called a rear camera device; it can also be installed on both the front and back of the electronic device 100, as shown in Figure 1, where the camera device 40 is installed on both the front and back of the electronic device 100, so that it can capture the scene located on one side of the front of the electronic device 100 and the scene located on one side of the back of the electronic device 100, as long as the appropriate camera device is used when shooting.

[0067] It should be understood that the installation position of the camera device 40 is merely illustrative. In some embodiments, when the camera device 40 is used as a front-facing camera, it can also be installed in other positions on the electronic device 100, such as on the left side of the earpiece, the upper center of the electronic device 100, the lower part of the electronic device 100, or one of the four corners of the electronic device 100; when the camera device 40 is used as a rear-facing camera, it can be installed in the upper center or upper right corner of the back of the electronic device 100. In other embodiments, the camera device 40 may not be located on the main body of the electronic device 100, but rather on an edge protruding from the main body of the electronic device 100, or on a movable or rotatable component relative to the electronic device 100, such as a component that can extend, retract, or rotate from the main body of the electronic device 100. When the camera device 40 can rotate relative to the electronic device 100, the camera device 40 functions as both a front-facing camera and a rear-facing camera, meaning that by rotating the same camera device 40, it can capture images from both the front and rear sides of the electronic device 100. In other embodiments, when the display screen 20 can be folded, the camera device 40 can be used as either a front camera or a rear camera. The camera device 40 is used to capture the view on the front side of the electronic device 100 or the view on the back side of the electronic device 100 as the display screen 20 is folded.

[0068] This application embodiment does not limit the number of camera devices 40; there can be one, two, three, or even more. For example, the electronic device 100 can have one or more camera devices 40 on the front or one or more camera devices 40 on the back. This application embodiment does not limit the number of camera devices, nor does it limit the relative positions of multiple camera devices. When multiple camera devices 40 are provided, they can be identical or different. For example, the number of lenses included in the multiple camera devices 40 may differ, or the optical parameters of the lenses may differ, or the positions of the lenses may differ, etc.

[0069] The camera device 40 can be used to capture videos and / or photos, and can be used to capture scenes at different distances. For example, the camera device 40 can be used to capture distant scenes, close-up scenes, and macro scenes. This application does not impose any special limitations on the embodiments.

[0070] Optionally, the electronic device 100 may further include a lens protection lens 103 for protecting the camera device 40. The lens protection lens 103 is disposed on the housing 10 and is used to cover the camera device 40. When the lens protection lens 103 is used to protect the front camera device, the lens protection lens 103 may cover only the front camera device or cover the entire front of the electronic device 100. When the lens protection lens 103 covers the entire front of the electronic device 100, it can simultaneously protect the front camera device and the display screen 20. The lens protection lens 103 is a cover glass (CG). When the lens protection lens 103 is used to protect the rear camera device, the lens protection lens 103 may cover the entire back of the electronic device 100 or may be disposed only at the position corresponding to the rear camera device to protect the rear camera device. The material of the lens protection lens 103 may be glass, sapphire, ceramic, etc., and this embodiment does not impose any special limitations. In some embodiments, the lens protection lens 103 is transparent, thereby allowing light beams from outside the electronic device 100 to pass through the lens protection lens 103 and enter the camera device 40.

[0071] It should be noted that the front of the electronic device 100 described in this application embodiment can be understood as the side surface of the electronic device 100 facing the user when the user uses the electronic device 100, and the back of the electronic device 100 can be understood as the side surface of the electronic device 100 facing away from the user when the user uses the electronic device 100.

[0072] It should be understood that the electronic device 100 shown in Figure 1 is not limited to the above-mentioned devices, but may also include other devices, such as batteries, flashlights, fingerprint recognition modules, earpieces, buttons, sensors, etc. This application embodiment only uses an electronic device with a camera device 40 installed as an example for illustration, but the components installed on the electronic device 100 are not limited to this.

[0073] In some embodiments, the electronic device 100 may further include an analog-to-digital converter (also known as an A / D converter, not shown in the figure). The analog-to-digital converter is connected between the camera device 40 and the image processor 30. The analog-to-digital converter is used to convert the signal generated by the camera device 40 into a digital image signal and transmit it to the image processor 30. The image processor 30 then processes the digital image signal and finally displays the image or video on the display screen 20.

[0074] In some embodiments, the electronic device 100 may further include a memory (not shown) communicatively connected to the image processor 30. The image processor 30 processes the digital image signal and then transmits the image to the memory so that the image can be retrieved from the memory and displayed on the display screen 20 at any time when it is needed to view the image later. In some embodiments, the image processor 30 may also compress the processed digital image signal before storing it in the memory to save memory space.

[0075] As cameras are increasingly used in electronic devices, the demands on image quality are also rising. Optical image stabilization (OIS) is an optical image stabilization technology that uses a movable lens assembly mounted on the camera lens or image sensor to counteract camera shake when shooting with a handheld electronic device, thereby reducing image blur and out of focus. However, current OIS technology faces the following challenges:

[0076] 1. Due to hardware limitations and the presence of noise, it is difficult to handle real-world jitter. Therefore, as the exposure time increases, the image remains blurry.

[0077] 2. Due to the low frame rate of traditional cameras, it is difficult to track fast-moving objects and acquire motion vectors in a short time. Therefore, when the subject moves quickly, a clear image cannot be achieved. How to achieve clear imaging with cameras is a problem that urgently needs to be solved.

[0078] To address the aforementioned issues, this application provides a camera device with a dynamic vision sensor (DVS), its control method, and an electronic device. Due to the high dynamic range and low latency characteristics of the DVS sensor, this camera device can enhance OIS (Optical Image Stabilization) performance by combining the characteristics of the DVS sensor, effectively improving the image clarity, especially in low-light, long-exposure scenarios. Furthermore, this camera device can acquire the motion vector of a moving object in an extremely short time and drive the motor in real-time, thereby improving the image clarity when shooting high-speed moving objects.

[0079] Figure 2 is a schematic diagram of a camera device provided in an embodiment of this application.

[0080] As shown in Figure 2(a), the camera device 40 may include a first camera module 41, which includes a first lens group 411, a first image stabilization element 412, and a first imaging sensor 413. The first imaging sensor 413 is located on the image side of the first lens group 411. The first camera module 41 may also include a circuit board (not shown in the figure), and the first imaging sensor 413 may be fixed to the circuit board. A light beam can pass through the first lens group 411 and illuminate the photosensitive unit of the first imaging sensor 413. The first image stabilization element 412 can drive the components in the first camera module 41 to move. For example, it can drive the first lens group 411 to move, or it can drive the first imaging sensor 413 to move, or it can drive the first lens group 411 and the first imaging sensor 413 to move simultaneously.

[0081] It is understood that the first lens group 411 may include one or more lenses, one or more prisms, or other optical elements that utilize the principles of optical refraction or optical reflection. When the first lens group 411 includes a prism, the first camera module 41 may be a periscope camera module. The embodiments of this application do not limit the arrangement of optical elements in the lens group.

[0082] In some possible implementations, the camera device 40 also includes a second image stabilization element 42, which can drive the first camera module 41 to move.

[0083] As shown in Figure 2(b), the camera device 40 may further include multiple camera modules. For example, the camera device 40 may also include a second camera module 43, which includes a second lens group 431, a first image stabilization element 432, and a second imaging sensor 433. The second imaging sensor 433 is located on the image side of the second lens group 431. The second camera module 43 may also include a circuit board (not shown in the figure), and the second imaging sensor 433 may be fixed to the circuit board. The light beam can pass through the second lens group 431 and illuminate the photosensitive unit of the second imaging sensor 433. The first image stabilization element 432 can drive the components in the second camera module 43 to move. For example, it can drive the second lens group 431 to move, or it can drive the second imaging sensor 433 to move, or it can drive the second lens group 431 and the second imaging sensor 433 to move simultaneously. The camera device 40 also includes a second image stabilization element 42, which can drive multiple camera modules to move. For example, the second image stabilization element 42 can drive the first camera module 41 and the second camera module 43 to move simultaneously.

[0084] Similarly, the second lens group 431 may include one or more lenses, one or more prisms, or other optical elements that utilize the principles of optical refraction or optical reflection. When the second lens group 431 includes a prism, the second camera module 43 may be a periscope camera module. The embodiments of this application do not limit the arrangement of optical elements in the lens group.

[0085] It should be noted that in Figure 2(b), the first camera module 41 and the second camera module 43 may be the same or different. For example, the camera modules in the camera device 40 may be divided into a main camera module, a telephoto module, a wide-angle module, etc. The first camera module 41 and the second camera module 43 may be the same camera module among the three types of camera modules, or they may be different camera modules. The same camera module means that the lens group in the camera module is set in the same way and has similar or the same optical performance. The embodiments of this application do not limit this.

[0086] The first image stabilization element 412 may include an optical image stabilization (OIS) motor. OIS motors can be divided into various types depending on the driving component, such as lens OIS, sensor OIS, prism OIS, etc.

[0087] The second image stabilization element 42 can be a gimbal optical image stabilization motor (OIS). The OIS can drive the first camera module 41 to move as a whole, and the OIS can also drive the first camera module 41 and the second camera module 42 to move simultaneously.

[0088] Lens OIS refers to the movement of the lens or lens group in the first lens group 411 of the first camera module 41 when the OIS motor is working.

[0089] Sensor OIS refers to the OIS motor driving the first imaging sensor 413 in the first camera module 41 to move when it is working, such as driving the DVS sensor to move.

[0090] OIS refers to the movement of the prism or prism group in the first lens group 411 of the first camera module 41 when the OIS motor is working.

[0091] In some possible implementations, the above-mentioned methods of driving different components with OIS motors can be used in parallel. For example, lens+sensor OIS means that when the OIS motor is working, it drives the first lens group 411 and the first imaging sensor 413 in the first camera module 41.

[0092] Similarly, the relevant settings and characteristics of the first image stabilization element 432 in the second camera module 43 can be referred to the description of the first image stabilization element 412 above, and will not be repeated here for the sake of brevity.

[0093] It should be noted that in Figure 2(b), the components driven by the first image stabilization element 412 and the first image stabilization element 432 during operation can be the same or different. For example, when the first image stabilization element 412 drives the first lens group 411 to move, the first image stabilization element 432 can drive the second lens group 431 to move or drive the second imaging sensor 433 to move. In other words, in a camera device with multiple camera modules, the driving mode of the first image stabilization element in each camera module is set independently.

[0094] It should be noted that when the camera device 40 includes multiple camera modules, in order to ensure the uniformity of the image, the electronic device needs to drive the second image stabilization element 42 to move, so that the first camera module 41 and the second camera module 43 can move synchronously, driving the multiple camera modules in the camera device 40 to move together, thereby not affecting the imaging effect.

[0095] It is understood that the driving methods of the first image stabilization element 412 and the second image stabilization element 42 include, but are not limited to, the above-mentioned driving methods and combinations thereof. For example, the first image stabilization element 412 and the second image stabilization element 42 can move simultaneously to achieve the image stabilization effect.

[0096] It should be noted that the camera device 40 shown in Figure 2(b) may also include other camera modules besides the first camera module 41 and the second camera module 43. The number of camera modules in the camera device 40 is not limited in this embodiment of the application; it may be one, two, three, or more. The relevant characteristics and settings of these camera modules can be referred to the description of the first camera module 41 or the second camera module 43 above. For the sake of brevity, they will not be repeated here.

[0097] The imaging sensor can include a CIS sensor (CMOS image sensor) or a DVS sensor. A CIS sensor (also known as a photosensitive element) is a semiconductor chip with hundreds of thousands to millions of photodiodes on its surface, which generate charges when exposed to light. A CIS sensor can also be a complementary metal-oxide semiconductor (CMOS) device. CMOS devices primarily utilize silicon and germanium to create semiconductors where N-type (negative) and P-type (positive) semiconductors coexist. The current generated by these complementary effects can be recorded and interpreted by the processing chip as an image.

[0098] DVS sensors, also known as event cameras, differ from CIS sensors used for image acquisition. DVS can be simply understood as a sensor that "only senses moving objects." Each pixel in a DVS sensor has an independent photoelectric sensing module. When the brightness change at that pixel exceeds a set threshold, event information (sometimes called pulse data) is generated and output. Furthermore, because all pixels operate independently, the data output of an event camera is asynchronous and spatially sparse. This is the biggest difference between DVS sensors and conventional image sensors. DVS sensors are event-driven photoelectric sensors. Each pixel in a DVS sensor independently senses changes in light intensity, and pixels with light intensity changes exceeding a threshold are considered active pixels. The row and column position, polarity position, timestamp, and other information of the active pixels are then packaged, encoded, and output in real time.

[0099] The first imaging sensor 413 includes a first DVS sensor, and the second imaging sensor 433 includes a second CIS sensor. The first imaging sensor 413 may also include the first CIS sensor, and the second imaging sensor 433 may also include the second DVS sensor.

[0100] DVS and CIS sensors can be configured together or separately.

[0101] A fusion-configuration imaging sensor refers to an imaging sensor whose photosensitive unit can collect both red, green, and blue (RGB) pixels and DVS pixels; or, a portion of the photosensitive unit of the imaging sensor is used to collect RGB pixels, and another portion is used to collect DVS pixels.

[0102] Here, RGB pixels refer to the pixels required for the CIS sensor to output image information, and DVS pixels refer to the pixels required for the DVS sensor to output event information to provide brightness change information.

[0103] Figure 3 is a schematic diagram of an imaging sensor provided in an embodiment of this application.

[0104] Figure 3(a) shows a schematic diagram of a fusion imaging sensor. A fusion imaging sensor can share pixels on the same integrated circuit (IC) chip, simultaneously acquiring RGB pixels and DVS pixels, and outputting image information and event information. Figure 3(b) shows a schematic diagram of another fusion imaging sensor. In a fusion imaging sensor, the IC chip can also not share pixels. On each photosensitive unit, some pixels are used to acquire DVS pixels, and other pixels are used to acquire RGB pixels.

[0105] Figure 3(c) shows a schematic diagram of a separately configured imaging sensor. A separately configured imaging sensor can mean that the CIS sensor and the DVS sensor operate independently. For example, a DVS sensor can be placed around the CIS sensor. The CIS sensor and the DVS sensor are controlled by two different IC chips. The photosensitive unit on the CIS sensor can only acquire RGB pixels and output image information, while the photosensitive unit on the DVS sensor can only acquire DVS pixels and output event information or event data. Figure 3(c) shows a total of four DVS sensors arranged around the CIS sensor. It should be understood that Figure 3(c) is only a schematic diagram of one possible arrangement of the CIS sensor and DVS sensor. In some possible implementations, one DVS sensor can be placed around the CIS sensor to achieve event information output, or multiple DVS sensors can be placed. The DVS sensors can be placed at any position around the CIS sensor. The embodiments of this application do not limit the position or number of DVS sensors around the CIS sensor.

[0106] It should be noted that the setting method of the DVS sensor and CIS sensor provided in this application embodiment can be applied to both the first imaging sensor 413 and the second imaging sensor 433.

[0107] In some possible implementations, when the first imaging sensor 413 is configured as the first CIS sensor, the second imaging sensor 433 can be configured as the second DVS sensor; or, when the first imaging sensor 413 is configured as the first DVS sensor, the second imaging sensor 433 can be configured as the second CIS sensor.

[0108] Figure 4 is a schematic flowchart of a control method S300 for a camera device provided in an embodiment of this application.

[0109] This control method can be applied to electronic devices equipped with a camera device 40.

[0110] S310, determine the target motion vector.

[0111] The target motion vector refers to the motion vector value that an electronic device hopes to achieve in the captured image after image stabilization.

[0112] In some possible implementations, the target motion vector can be set to 0. When the target motion vector is 0, it can be assumed that after the camera device 40 has performed image stabilization, the objects in the captured image have not been blurred or moved, and the image is clear.

[0113] S320 determines the feedback motion vector in the captured image.

[0114] When a user takes a picture using an electronic device, the DVS sensor can acquire event information. The electronic device calculates the feedback motion vector based on the event information, thereby determining the feedback motion vector in the captured image.

[0115] In some possible implementations, electronic devices can also calculate the feedback motion vector in the captured image based on the image information output by the CIS sensor and the event information acquired by the DVS sensor. This is because the event information includes information on pixel brightness changes, which cannot reflect the details in the captured image, while the image information output by the CIS sensor has richer image details and textures. Adding the image information output by the CIS sensor to the calculation of the feedback motion vector can improve the accuracy of the calculated feedback motion vector, thereby improving the image stabilization effect.

[0116] In some possible implementations, step S320 may also include S321, determining the source of the feedback motion vector.

[0117] Understandably, when a user takes a picture using an electronic device, the captured image can usually be divided into a foreground image and a background image. The feedback motion vector in the foreground image can be called the foreground feedback motion vector, and the feedback motion vector in the background image can be called the background feedback motion vector. Because the DVS sensor has a high dynamic range (120dB) and low latency (<1ms), the electronic device can distinguish between the feedback motion vector in the foreground image and the feedback motion vector in the background image based on the event information output by the DVS. The electronic device can achieve different optical image stabilization effects for different applications based on the different sources of the feedback motion vectors, or in other words, determine the source of the feedback motion vector used to drive the movement of the image stabilization element. For example, when the feedback motion vector comes from a foreground object, it can solve the problem of blurry subject when tracking; when the feedback motion vector comes from a background object, it can solve the problem of shaking during shooting.

[0118] When the feedback motion vector is the foreground feedback motion vector, the anti-shake element is driven to move, and the background feedback motion vector is saved.

[0119] In this step, when the user takes a picture using an electronic device, it is determined that the foreground object needs to be stabilized. However, in the subsequent processing, the user may also need to stabilize the background object. Therefore, it is necessary to save the background feedback motion vector in the captured image so that the background feedback motion vector can be further processed in the subsequent process, thereby achieving the effect of panoramic image stabilization in the captured image.

[0120] Electronic devices perform image stabilization on background feedback motion vectors. In some possible implementations, image stabilization on background feedback motion vectors can be achieved through deblurring, for example, using deblurring methods based on deep learning and neural networks.

[0121] When the feedback motion vector is the background feedback motion vector, the anti-shake element is driven to move, and the foreground feedback motion vector is saved;

[0122] In this step, when the user takes a picture using an electronic device, it is determined that the background object needs to be stabilized. However, in the subsequent processing, the user may also need to stabilize the foreground object. Therefore, it is necessary to save the foreground feedback motion vector in the captured image so that the foreground feedback motion vector can be further processed in the subsequent process, thereby achieving the effect of panoramic image stabilization in the captured image.

[0123] Electronic devices perform stabilization on foreground feedback motion vectors. In some possible implementations, stabilization of the foreground feedback motion vectors can be achieved through deblurring, for example, using deblurring methods based on deep learning and neural networks.

[0124] Based on the above technical solutions, electronic devices can perform image stabilization based on foreground feedback motion vectors or background feedback motion vectors respectively, thereby enabling applications in different scenarios. They can also achieve image stabilization for the entire panorama of the captured image by combining foreground feedback motion vectors and background feedback motion vectors.

[0125] The S330 drives the image stabilization element to move based on the feedback motion vector.

[0126] In order for electronic devices to achieve image stabilization, the electronic devices need to determine the feedback motion vector in the captured image through the camera device, and then drive the image stabilization element to move according to the feedback motion vector, thereby driving the components in the camera module to move or driving the camera module to move.

[0127] After determining the feedback motion vector, the feedback motion vector is projected onto the x and y directions, thereby controlling the anti-shake element to move in the opposite direction of the projection of the feedback motion vector.

[0128] In some possible implementations, the x and y directions of the projection of the feedback motion vector can be distinguished according to the movement mode of the electronic device. For example, when the electronic device moves in a translation mode, the x and y directions can be from left to right and from top to bottom, respectively; when the electronic device moves in a rotation mode, the x and y directions can be the yaw direction and the pitch direction, respectively.

[0129] In some possible implementations, the method further includes step S331, driving the stabilization element to move based on the difference between the feedback motion vector and the target motion vector.

[0130] Once the feedback motion vector is determined, the electronic device can determine the difference Δ between the feedback motion vector and the target motion vector, and project Δ onto the x and y directions, thereby controlling the movement of the anti-shake element more precisely.

[0131] It is understandable that when controlling the movement of the image stabilization element, the amount of movement determined by the electronic device can be k times the value of the difference Δ component mentioned above, where k is a coefficient value, and the value of k can be calibrated according to the specific parameters of the camera module.

[0132] In the above steps, the image stabilization element driven by the electronic device can be the first image stabilization element 412 in Figure 2(a), or it can be the second image stabilization element 42, or the first image stabilization element 412 and the second image stabilization element 42 can be driven simultaneously. Similarly, in Figure 2(b), the first image stabilization element 412 and the second image stabilization element 42 can be driven separately or simultaneously. The relevant descriptions and settings of the first image stabilization element 412 and the second image stabilization element 42 can be referred to the description of Figure 2 above, and will not be repeated here for the sake of brevity.

[0133] It should be noted that when the camera device has multiple camera modules, such as shown in Figure 2(b), the electronic device needs to drive the second image stabilization element 42 to move so that the first camera module 41 and the second camera module 43 can move synchronously, so as not to affect the imaging effect.

[0134] In some possible implementations, when the second image stabilization element 42 is driven, the first image stabilization element 412 in the first camera module 41 or the first image stabilization element 432 in the second camera module 43 can also be driven simultaneously to achieve the image stabilization effect.

[0135] The relevant descriptions and settings of the first image stabilization element 412, the first image stabilization element 432, and the second image stabilization element 42 can be found in the description of Figure 2 above. For the sake of brevity, they will not be repeated here.

[0136] S340: When the feedback motion vector is the same as the target motion vector, output an image.

[0137] When the electronic device drives the image stabilization element to move, the feedback motion vector in the captured image is also constantly changing. Therefore, the electronic device continuously acquires the feedback motion vector in the captured image based on the event information and image information output by the camera device, and continuously adjusts the image based on the feedback motion vector. When the feedback motion vector in the captured image is the same as the target motion vector at a certain moment, it can be considered that the sharpness of the captured image has met the target image stabilization requirements, and the image of the captured image can be output.

[0138] In some possible implementations, the imaging sensor in the camera module includes only a DVS sensor. In this case, the camera device determines the motion vector in the captured image based on cumulative frames of preceding and following events or an event-by-event stream.

[0139] In some possible implementations, the imaging sensors in the camera module include DVS sensors and CIS sensors. In this case, the method for determining the feedback motion vector before exposure is different from the method for determining the feedback motion vector during exposure.

[0140] When the electronic device images before exposure, it determines the feedback motion vector based on the preview stream image and event information, including event accumulation frames or event-by-event streams.

[0141] When the electronic device is in the exposure period, the feedback motion vector is determined based on the event information.

[0142] Preview stream images refer to image data output by the camera module after acquiring light signals and processing them through the CIS sensor. This image data can be sent to the display unit of the electronic device in the form of a data stream and finally displayed in the image preview area of ​​the electronic device's screen.

[0143] Figure 5 is a schematic diagram of the working principle of the DVS sensor.

[0144] Referring to Figure 5, the event accumulation frame and event-by-event stream are explained. As shown in Figure 5, x and y represent the x-axis and y-axis of the current captured image, and the t-axis represents time. As time changes, when the disk rotates clockwise, the position of the black dots on the disk changes, resulting in a change in pixel brightness. At this time, after the DVS sensor receives the pixel brightness change in the current captured image, it generates a series of events and outputs them in the form of an event-by-event stream. These events have three elements: timestamp, pixel coordinates, and polarity, which respectively indicate "at what time, which pixel point, the brightness increased or decreased." The event accumulation frame refers to the image obtained by projecting the 3D point cloud of the event-by-event stream along the t-axis dimension onto the xy-plane. The 3D point cloud refers to a dataset of 3D coordinate points arranged according to a regular grid.

[0145] Before exposure, electronic devices take images. This can be understood as the electronic device taking pictures before the shutter is pressed and before the exposure state is entered. The CIS sensor continues to receive light signals and continuously output image data. This continuously output image data is displayed on the electronic device's screen and is called a preview stream image.

[0146] Understandably, since the DVS sensor can only receive brightness change information in the current subject image, the image details in the subject image are very limited. Therefore, the feedback motion vector in the subject image can be calculated by combining the image details in the preview stream image output by the CIS sensor and the event information acquired by the DVS sensor, thereby achieving a more accurate image stabilization effect.

[0147] When an electronic device is in the exposure period, it can be understood that the electronic device has already pressed the shutter and entered the exposure state when taking a picture. At this time, although the CIS sensor is still receiving light signals, it has stopped outputting image data, so there is no preview stream image output. Therefore, the feedback motion vector in the subject image can only be calculated based on the event information output by the DVS sensor.

[0148] The embodiments of this application also provide an electronic device, which includes any of the camera devices described in the above embodiments, and the electronic device can be used to execute the control method of the camera device.

[0149] Figure 6 is a schematic block diagram of another electronic device provided in an embodiment of this application. As shown in Figure 6, the electronic device 200 includes an image processor 30 and a camera device 40. The image processor 30 is communicatively connected to the camera device 40. The image processor 30 may include a motion vector detection module 31, a stabilization control module 32, and an image segmentation module 33. The motion vector detection module 31 can determine the feedback motion vector based on the image information and / or event information output by the camera device 40. The stabilization control module 32 drives the stabilization element in the camera device 40 to move according to the feedback motion vector to achieve the stabilization effect. The image segmentation module 33 can divide the feedback motion vector into foreground feedback motion vector and background feedback motion vector according to the source of the feedback motion vector in the captured image, so that the stabilization control module 32 drives the stabilization element in the camera device 40 to move according to the foreground feedback motion vector or the background feedback motion vector to achieve different stabilization applications. In some possible implementations, the motion vector detection module 31 can be used to perform steps S310 and S320 in the method S300 provided in the embodiments of this application, the image stabilization control module 32 can be used to perform steps S330 and S331, the image segmentation module 33 can be used to perform step S321, and the image processor 30 can be used to perform step S340.

[0150] Embodiments of this application provide an electronic device, including any of the camera devices described in the above embodiments, one or more processors, one or more memories, and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the control method provided in the embodiments of this application.

[0151] Embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed, implement the control method provided in the embodiments of this application.

[0152] An embodiment of this application provides a chip that stores instructions, which, when executed, implement the control method provided in the embodiment of this application.

[0153] The embodiments of this application provide a computer program product, which stores a program or instructions. When the program or instructions are run, the control method provided in the embodiments of this application is implemented.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0159] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method for a camera device, characterized in that, The camera device includes an imaging sensor and an image stabilization element, the imaging sensor including a dynamic vision sensor (DVS), and the method includes: Determine the target motion vector; The feedback motion vector in the captured image is determined by the Dynamic Visual Sensor (DVS). The anti-shake element is driven to move according to the feedback motion vector; An image is output when the feedback motion vector is the same as the target motion vector.

2. The method according to claim 1, characterized in that, The step of driving the image stabilization element to move according to the feedback motion vector includes: The motion of the anti-shake element is controlled according to the opposite direction of the projection direction of the feedback motion vector.

3. The method according to claim 1 or 2, characterized in that, The step of driving the image stabilization element to move according to the feedback motion vector includes: The anti-shake element is driven to move based on the difference between the feedback motion vector and the target motion vector.

4. The method according to any one of claims 1-3, characterized in that, Determining the feedback motion vector in the captured image includes: The feedback motion vector is determined based on event information, which includes event accumulation frames or event-by-event streams.

5. The method according to claim 4, characterized in that, Determining the feedback motion vector in the captured image includes: When the camera is in the exposure period, it determines the feedback motion vector based on the event information.

6. The method according to any one of claims 1-5, characterized in that, The imaging sensor also includes an image sensor (CIS), which is used to determine image information in the captured image. When the camera device images before exposure, determining the feedback motion vector in the captured image includes: determining the feedback motion vector based on the preview stream image and event information, wherein the event information includes event accumulation frames or event-by-event streams.

7. The method according to any one of claims 1-5, characterized in that, The step of driving the image stabilization element to move according to the feedback motion vector further includes: Determine the source of the feedback motion vector; The feedback motion vector includes a foreground feedback motion vector or a background feedback motion vector; The stabilization element is driven to move based on the source of the feedback motion vector.

8. The method according to claim 7, characterized in that, Based on the source of the feedback motion vector, the stabilization element is driven to move, including: When the feedback motion vector is the foreground feedback motion vector, the stabilization element is driven to move, and the background feedback motion vector is saved.

9. The method according to claim 8, characterized in that, The method further includes: The background feedback motion vector is subjected to anti-shake processing.

10. The method according to claim 7, characterized in that, Based on the source of the feedback motion vector, the stabilization element is driven to move, including: When the feedback motion vector is the background feedback motion vector, the stabilization element is driven to move, and the foreground feedback motion vector is saved.

11. The method according to claim 10, characterized in that, The method further includes: The foreground feedback motion vector is subjected to anti-shake processing.

12. A camera device, characterized in that, The camera device includes a first camera module, which includes a first lens group, a first imaging sensor, and a first image stabilization element. The first imaging sensor includes a first dynamic vision sensor (DVS). The first lens group is used to collect incident light onto the photosensitive unit of the first imaging sensor; The first dynamic vision sensor (DVS) is used to determine the feedback motion vector in the captured image; The first image stabilization element is used to drive the components in the first camera module to move according to the feedback motion vector.

13. The camera device according to claim 12, characterized in that, The camera device also includes a second image stabilization element. The second image stabilization element is used to drive the camera module in the camera device to move according to the feedback motion vector.

14. The camera device according to claim 13, characterized in that, The second image stabilization element includes a gimbal optical image stabilization motor.

15. The camera device according to any one of claims 12-14, characterized in that, The first imaging sensor also includes a first image sensor (CIS). The first image sensor (CIS) is used to acquire image information from the captured image.

16. The camera device according to any one of claims 12-15, characterized in that, The first image stabilization element includes at least one of a sensor optical image stabilization motor, a lens optical image stabilization motor, or a prism optical image stabilization motor.

17. The camera device according to any one of claims 12-16, characterized in that, The camera device further includes a second camera module, which includes a second lens group, a second imaging sensor, and the first image stabilization element. The second imaging sensor includes at least a second image sensor (CIS). The second lens group is used to collect incident light onto the photosensitive unit of the second imaging sensor. The second image sensor (CIS) is used to acquire image information in the captured image; the first image stabilization element of the second camera module is used to drive the components in the second camera module to move according to the feedback motion vector.

18. The camera device according to claim 17, characterized in that, The second imaging sensor also includes a second dynamic vision sensor (DVS). The second dynamic vision sensor (DVS) is used to determine the feedback motion vector in the captured image.

19. An electronic device, characterized in that, The electronic device includes a camera device as described in any one of claims 12-18. One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-11.

20. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 1-11.

21. A chip, characterized in that, The chip stores instructions that, when executed, implement the method as described in any one of claims 1-11.

22. A computer program product, characterized in that, The computer program product stores a program or instructions that, when executed, implement the method as described in any one of claims 1-11.

Images