Shooting method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a shooting method, electronic equipment, a storage medium and a program product, and relates to the technical field of light imaging, and the method comprises the steps: carrying out the motion prediction of a target object based on a first image frame containing the target object, and one or more second image frames which are shot before the first image frame and contain the target object, obtaining a prediction frame of the target object in the third image frame corresponding to the target shooting moment and depth information corresponding to the prediction frame; the target shooting moment is later than the shooting moment of the first image frame; and focusing based on the prediction frame and the depth information, and shooting a third image frame at the target shooting moment through the focal length after focusing. By applying the embodiment of the invention, the definition during image shooting can be improved.

Description

Technical Field

[0001] The present application relates to the field of optical imaging technology, and in particular to a shooting method, electronic equipment, storage medium and program product. Background Art

[0002] During the shooting process, when the subject is a fast-moving object or person, since the subject is in motion, it is difficult for the auto focus (AF) algorithm to accurately focus on the subject, resulting in unclear subject in the imaging result, which will cause a bad experience for users. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a shooting method, electronic device, storage medium and program product to improve the clarity of the captured image. The specific technical solution is as follows:

[0004] In a first aspect, an embodiment of the present application provides a shooting method, including:

[0005] Based on a first image frame including a target object and one or more second image frames including the target object captured before the first image frame, motion prediction is performed on the target object to obtain a predicted frame of the target object and depth information corresponding to the predicted frame in a third image frame corresponding to a target capturing time, the target capturing time being later than a capturing time of the first image frame;

[0006] Focusing is performed based on the predicted frame and the depth information, and the third image frame is captured at the target shooting moment with the focused focal length.

[0007] In one embodiment of the present application, the target object in the first image frame is determined based on the following method:

[0008] Obtaining depth maps corresponding to the first image frame and the second image frame;

[0009] Based on the depth information variation in the depth map, a moving object in the first image frame is determined, and the moving object is used as the target object.

[0010] In one embodiment of the present application, the step of determining a moving object in the first image frame based on the depth information variation in the depth map and taking the moving object as the target object includes:

[0011] Based on the depth information variation in the depth map, the moving objects in the first image frame and the moving speed of each of the moving objects are determined, and the moving object with the fastest moving speed is determined as the target moving object.

[0012] In one embodiment of the present application, before performing motion prediction on the target object based on a first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame, the method further includes:

[0013] Image data collected by the sensor is acquired, and timestamps of the image data are aligned through time soft synchronization to obtain image data corresponding to the first image frame / the second image frame.

[0014] In one embodiment of the present application, the step of performing motion prediction on the target object based on a first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame to obtain a predicted frame of the target object in a third image frame corresponding to the target shooting moment and depth information corresponding to the predicted frame includes:

[0015] Determine the three-dimensional position coordinates of the target object in the first image frame and the second image frame based on the detection frames of the target object in the first image frame and the second image frame and the corresponding depth information, and determine the three-dimensional motion speed of the target object based on the three-dimensional position coordinates of the target object in the first image frame and the second image frame;

[0016] Based on the three-dimensional motion speed and the preset motion equation, the three-dimensional position coordinates of the target object at the target shooting moment are determined, and the prediction frame of the target object in the third image frame corresponding to the target shooting moment and the depth information corresponding to the prediction frame are obtained.

[0017] In one embodiment of the present application, the step of focusing based on the prediction frame and the depth information, and capturing the third image frame at the target shooting moment with the focused focal length includes:

[0018] A target standard focus position is calculated based on the depth information corresponding to the prediction frame, and the lens is driven to move to the target standard focus position so that the prediction frame is focused on the imaging plane.

[0019] In a second aspect, an embodiment of the present application provides a photographing device, including:

[0020] a prediction module, configured to perform motion prediction on the target object based on a first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame, to obtain a prediction frame of the target object and depth information corresponding to the prediction frame in a third image frame corresponding to a target capturing time, wherein the target capturing time is later than a capturing time of the first image frame;

[0021] A focusing module is used to focus based on the prediction frame and the depth information, and to capture the third image frame at the target shooting moment with a focused focal length.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute any one of the steps described in the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described in the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the methods described in the first aspect.

[0025] Beneficial effects of the embodiments of the present application:

[0026] The shooting method provided in the embodiment of the present application predicts the motion of the target object by using the first image frame and the second image frame containing the target object, obtains the prediction frame and the depth information corresponding to the prediction frame in the third image frame corresponding to the target shooting time, focuses based on the prediction frame and the depth information corresponding to the prediction frame, and shoots the third image frame at the target shooting time with the focal length after focusing. Among them, the prediction frame and the depth information corresponding to the prediction frame obtained based on the motion prediction are closer to the real ROI and the corresponding real depth information of the target object in the third image frame than the ROI and the corresponding depth information of the target object in the current image frame that has been shot, that is, the deviation between the predicted ROI and the intended focus area is smaller. Therefore, when focusing on the third image frame before the target shooting time, the prediction frame and the depth information corresponding to the target object are used to replace the ROI and the corresponding depth information of the target object in the current image frame that has been shot by the conventional focusing algorithm. The focusing accuracy is higher and the image captured by the target object is clearer. When the target object is specifically an object in a moving state, the image blur problem caused by the moving object can be avoided, and the shooting quality is improved.

[0027] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0029] Figure 1a A schematic diagram of the first hardware structure of the electronic device provided in this application;

[0030] Figure 1b The software structure diagram of the electronic device provided for this application;

[0031] Figure 2 It is a schematic diagram of a blurred image taken for a moving object;

[0032] Figure 3 is a schematic diagram of a focusing process in the related art;

[0033] Figure 4 is a flowchart of a shooting method provided in an embodiment of the present application;

[0034] Figure 5a A schematic diagram of a first effect of the shooting method provided in an embodiment of the present application;

[0035] Figure 5b A schematic diagram of a second effect of the shooting method provided in an embodiment of the present application;

[0036] Figure 6 is a depth map provided by an embodiment of the present application;

[0037] Figure 7 is a schematic diagram of a process for determining a target object provided in an embodiment of the present application;

[0038] Figure 8 is a schematic diagram of the focusing process of the shooting method provided in an embodiment of the present application;

[0039] Fig. 9 It is a structural schematic diagram of a shooting device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0041] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first instruction and the second instruction are to distinguish different user instructions, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0042] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0043] The embodiments of the present application can be applied to terminals with communication functions, such as mobile phones, tablet computers, personal computers (PCs), personal digital assistants (PDAs), smart watches, netbooks, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, vehicle-mounted devices, smart cars, robots, smart glasses, smart TVs, etc.

[0044] For example, Figure 1a The schematic diagram of the structure of the terminal 100 is shown. The terminal 100 may include a processor 110, a display screen 120, a camera 130, an internal memory 140, a SIM (Subscriber Identification Module) card interface 150, a USB (Universal Serial Bus) interface 160, a charging management module 170, a battery management module 171, a battery 172, a sensor module 180, a mobile communication module 190, a wireless communication module 200, an antenna 1 and an antenna 2, etc. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, etc.

[0045] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0046] The processor 110 may include one or more processing units, for example: the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units may be independent components or integrated in one or more processors. In some embodiments, the terminal 100 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be set in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the terminal 100 in processing data or executing instructions.

[0047] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 160 is an interface that complies with the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 160 may be used to connect a charger to charge the terminal 100, and may also be used to transmit data between the terminal 100 and a peripheral device. The USB interface 160 may also be used to connect headphones to play audio through the headphones.

[0048] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0049] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 190, the wireless communication module 200, the modem processor and the baseband processor.

[0050] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0051] The terminal 100 implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0052] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include one or more display screens 120.

[0053] In some embodiments of the present application, when the display panel is made of materials such as OLED, AMOLED, FLED, etc., the above Figure 1a The display screen 120 in the embodiment can be bent. Here, the display screen 120 can be bent means that the display screen can be bent to any angle at any position and can be kept at the angle, for example, the display screen 120 can be folded in half from the middle to the left or right. It can also be folded in half from the middle to the top or bottom.

[0054] The display screen 120 of the terminal 100 may be a flexible screen. Currently, flexible screens have attracted much attention for their unique characteristics and great potential. Compared with traditional screens, flexible screens are flexible and bendable, which can provide users with a new interaction method based on the bendable characteristics, and can meet more user demands for the terminal. For a terminal equipped with a foldable display screen, the foldable display screen on the terminal can be switched between a small screen in a folded form and a large screen in an unfolded form at any time. Therefore, users are using the split-screen function on terminals equipped with foldable displays more and more frequently.

[0055] The terminal 100 can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor, etc., wherein the camera 130 includes a front camera and a rear camera.

[0056] The ISP is used to process the data fed back by the camera 130. For example, when shooting, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be set in the camera 130.

[0057] The camera 130 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red green blue (RGB), YUV or other format. In some embodiments, the terminal 100 may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0058] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0059] The video codec is used to compress or decompress digital video. The terminal 100 may support one or more video codecs. Thus, the terminal 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0060] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0061] The internal memory 140 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the above instructions stored in the internal memory 140, so that the terminal 100 performs the video segmentation method provided in some embodiments of the present application, as well as various applications and data processing. The internal memory 140 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the terminal 100. In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash memory (Universal Flash Storage, UFS), etc. In some embodiments, the processor 110 can execute the video segmentation method provided in the embodiment of the present application, as well as other applications and data processing by running instructions stored in the internal memory 140 and / or instructions stored in a memory provided in the processor 110.

[0062] The internal memory 140 can be used to store the relevant programs of the video segmentation method provided in the embodiment of the present application, and the processor 110 can be used to call the relevant programs of the video segmentation method stored in the internal memory 140 when displaying information to execute the video segmentation method of the embodiment of the present application.

[0063] The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, and the like.

[0064] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 120. There are many types of pressure sensors 180A, for example, they can be resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When the force acts on the pressure sensor 180A, the capacitance between the electrodes changes, and the terminal 100 determines the intensity of the pressure according to the change in capacitance. When the touch operation acts on the display screen 120, the terminal 100 detects the touch operation according to the pressure sensor 180A. The terminal 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on a short message application icon, an instruction to view the short message is executed; when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0065] The fingerprint sensor 180B is used to collect fingerprints. The terminal 100 can use the collected fingerprint characteristics to realize functions such as unlocking, accessing application locks, taking pictures, and answering calls.

[0066] The touch sensor 180C is also called a touch control device. The touch sensor 180C can be arranged on the display screen 120, and the touch sensor 180C and the display screen 120 form a touch screen, which is also called a touch control screen. The touch sensor 180C is used to detect a touch operation acting on or near it. The touch sensor 180C can pass the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 180C can also be arranged on the surface of the terminal 100, and is arranged at a different position from the display screen 120.

[0067] The ambient light sensor 180D is used to sense the ambient light brightness. The terminal 100 can adaptively adjust the brightness of the display screen 120 according to the perceived ambient light brightness. The ambient light sensor 180D can also be used to automatically adjust the white balance when shooting. The ambient light sensor 180D can also transmit the environment information of the device to the GPU.

[0068] The ambient light sensor 180D is also used to obtain the brightness, light ratio, color temperature, etc. of the environment in which the camera 130 captures images.

[0069] Taking the electronic device as a smart phone as an example, the shooting method in the embodiment of the present application can be Figure 1b The smartphone system architecture shown is implemented in Figure 1b The system architecture of a smartphone includes a kernel, a framework layer, and an application layer. The kernel includes a driver layer and a real-time operating system. The driver layer includes a GPU (graphics processor), a display driver (LCD driver in the figure), a TP driver (touch screen driver), buttons, etc. The real-time operating system includes interrupt management, task scheduling, and MEM (memory management). The framework layer includes basic system capabilities, underlying software services, hardware service capabilities, etc. The application layer includes shooting applications, display applications, system applications, communication applications, etc.

[0070] In actual shooting, when the subject moves quickly, the current autofocus algorithm may not be able to ensure the subject is clear. Figure 2 , Figure 2 It is a schematic diagram of a blurred image taken of a moving object, which specifically shows a doll thrown from the opposite direction. It can be seen that the moving doll is not clear in the image and the image effect is not good.

[0071] The main reason for this problem is that when the subject is in motion, the focus frame monitored for the object is different from the actual position of the object during shooting. Therefore, the position of the focus frame is not the focus area intended by the user. As a result, the focal length calculated by the autofocus algorithm is inaccurate, resulting in unclear captured images.

[0072] Figure 3 This is a schematic diagram of the focusing process in the related art. For ease of understanding, the following is combined with Figure 3 The focusing process in the related art is exemplified. Figure 3 After the sensor outputs the image for the Nth frame, the detection algorithm sends the ROI (Region Of Interest) information. The ROI of the Nth frame is transmitted to the AF algorithm through the channel. The AF algorithm calculates the in-focus position and sends the code (drive value) to the motor after the exposure is completed. Based on the code sent by the AF algorithm, the motor pushes the position of the lens until it is stable to achieve the focus of the camera, thereby ensuring that the image taken by the camera at the in-focus position is clear. Based on the above description, it can be seen that when shooting the image of the N+1th frame, the current motor position is the in-focus position calculated for the previous frame. Specifically, in the process of the AF algorithm calculating the in-focus position, the Nth frame has been output, that is, the in-focus position calculated based on the ROI of the Nth frame can only take effect in the N+1th frame. The calculated in-focus position is naturally delayed by one frame. Therefore, the in-focus position calculated for fast-motion scenes is inaccurate, affecting the focusing effect.

[0073] In order to solve the above problems, the present application provides a shooting method. Figure 4 is a flow chart of the shooting method provided in the embodiment of the present application, see Figure 4 , the method specifically comprises the following steps:

[0074] Step S401: Based on a first image frame including a target object and one or more second image frames including the target object captured before the first image frame, motion prediction is performed on the target object to obtain a prediction frame of the target object in a third image frame corresponding to a target capturing time and depth information corresponding to the prediction frame; the target capturing time is later than a capturing time of the first image frame.

[0075] In the embodiment of the present invention, the first image frame and the second image frame are specifically images that have been captured, and the second image frame is captured before the first image frame. As an example, the first image frame can also be understood as the current frame.

[0076] Among them, the first image frame and the second image frame can be multiple continuous image frames, for example, the nth frame is the first image frame and the n-1th frame is the second image frame, or the first image frame and the second image frame can also be multiple image frames with a certain interval, for example, the nth frame is the first image frame and the n-2th frame is the second image frame. The embodiment of the present invention does not limit the specific selection of the first image frame and the second image frame.

[0077] The third image frame is an image frame that has not been captured yet. As an example, the third image frame can be the next frame of the current frame. In practical applications, electronic devices usually capture images based on a certain time rule, such as a fixed sampling interval. Therefore, the target capture time corresponding to the third image frame can be determined on this basis. As an example, if the sampling interval is Δt, the time to capture the first image frame is t n Based on this, the target acquisition time for shooting the third image frame can be determined as t n +Δt.

[0078] In the embodiment of the present application, the target object can be understood as a subject that needs to be presented clearly, and the embodiment of the present application does not limit the scope of the target object. As an example, since moving objects are more likely to be blurred, the moving object in the image frame can be selected as the target object to improve the image clarity. For example, optical flow detection can be performed between the first image frame and the second image frame to determine the moving object therein as the target object.

[0079] In an embodiment of the present application, before actually capturing the third image frame, motion prediction of the target object is performed in advance based on the first image frame and the second image frame containing the target object to determine the prediction box of the target object in the third image frame and the depth information corresponding to the prediction box.

[0080] As an example, based on the captured first image frame and the second image frame, the target object can be tracked, the tracking frame of the target object in each image frame is determined, the two-dimensional position coordinates of the tracking frame in the image frame are determined, and the depth information corresponding to the target object in the image frame is detected. The embodiment of the present application does not limit the specific method of detecting the depth information. As an example, the depth information of the target object can be obtained based on a TOF (Time Of Flight) sensor.

[0081] As an example, the camera coordinate system where the camera is located can be used as a reference to represent the position parameters of the target object in an image frame by (x, y, z), where x and y are the positions of the tracking box in the image frame, and z is the depth information of the target object.

[0082] When the position parameters of the target object in the first image frame and the second image frame are known, at least two points in the historical motion trajectory of the target object can be determined, so that the future motion trajectory of the target object can be predicted on this basis, and the prediction box of the target object in one or more future image frames and the depth information corresponding to the prediction box can be determined.

[0083] In the embodiment of the present application, the specific method for estimating the three-dimensional motion of the target object can be determined based on the actual situation, and the embodiment of the present application does not limit the specific algorithm of motion prediction.

[0084] Step S402: focusing based on the predicted frame and the depth information, and capturing a third image frame at the target capturing time with the focused focal length.

[0085] In practical applications, to complete focusing, it is usually necessary to select a focus area, that is, ROI. After determining the ROI, the focus plane of the lens is adjusted according to the depth of the ROI. The focus plane of the lens is selected to the depth corresponding to the ROI to complete the focusing.

[0086] In an embodiment of the present invention, it is believed that the prediction frame of the target object determined based on the above step S401 and the depth information corresponding to the prediction frame are close to the real ROI and the corresponding real depth information of the target object in the third image frame. Therefore, when calculating the focus parameters required for shooting the third image frame, the prediction frame and the depth information corresponding to the prediction frame are used to replace the ROI and the corresponding depth information in the current image frame that have been captured, which overcomes the delay existing in the current focus algorithm. When the target object is specifically an object in motion, compared with the ROI and the corresponding depth information of the object in the current image frame that has been captured, the prediction frame and the corresponding ROI will be closer to the real ROI and the real depth information corresponding to the object at the target acquisition moment, and the quasi-focus position calculated on this basis is more accurate, so that the focusing effect is better and the captured image is clearer.

[0087] Specifically, after calculating the in-focus position based on the prediction frame and the depth information, the lens can be driven by a motor to adjust the focus plane of the lens to correspond to the depth of the ROI, thereby completing the focusing.

[0088] It should be understood that the first image frame, the second image frame and the third image frame in the embodiment of the present application are only used as an exemplary illustration. For each image frame that needs to be shot, the motion of the target object can be predicted based on the previous image frame, and the focus parameters required for shooting the image frame can be determined based on the predicted frame and the depth information corresponding to the predicted frame, and the focus can be completed based on the predicted focus parameters before the actual shooting.

[0089] The shooting method provided in the embodiment of the present application predicts the motion of the target object by using the first image frame and the second image frame containing the target object, obtains the prediction frame and the depth information corresponding to the prediction frame in the third image frame corresponding to the target shooting time, focuses based on the prediction frame and the depth information corresponding to the prediction frame, and shoots the third image frame at the target shooting time with the focal length after focusing. Among them, the prediction frame and the depth information corresponding to the prediction frame obtained based on the motion prediction are closer to the real ROI and the corresponding real depth information of the target object in the third image frame than the ROI and the corresponding depth information of the target object in the current image frame that has been shot, that is, the deviation between the predicted ROI and the intended focus area is smaller. Therefore, when focusing on the third image frame before the target shooting time, the prediction frame and the depth information corresponding to the target object are used to replace the ROI and the corresponding depth information of the target object in the current image frame that has been shot by the conventional focusing algorithm. The focusing accuracy is higher and the image captured by the target object is clearer. When the target object is specifically an object in a moving state, the image blur problem caused by the moving object can be avoided, and the shooting quality is improved.

[0090] Combine the following Figure 5a and Figure 5b The application scenario of the present application is further described. The shooting method provided in the embodiment of the present application can be applied to shooting moving objects. Figure 5a This is a schematic diagram of the first effect of the shooting method provided in the embodiment of the present application. Figure 5b This is a schematic diagram of a second effect of the shooting method provided in an embodiment of the present application. Figure 5a and Figure 5b The photographing effects of a volleyball moving rapidly toward the camera are respectively shown. By adopting the photographing method provided in the embodiment of the present application, the focusing accuracy of a moving object can be improved and image blur caused by the movement of the object can be reduced.

[0091] In one embodiment of the present application, the target object in the first image frame may be determined based on the following method:

[0092] Acquire a depth map corresponding to the first image frame and the second image frame;

[0093] Based on the depth information variation in the depth map, a moving object in the first image frame is determined, and the moving object is used as a target object.

[0094] Specifically, compared with static objects, moving objects are more likely to be blurred in the image. Therefore, if the current moving object is determined as the target object and the target object is focused, it will help improve the clarity of the image and ensure the image effect.

[0095] In an embodiment of the present application, a depth map corresponding to the first image frame and the second image frame can be obtained. By comparing the change in depth information between the depth maps, the currently existing moving object can be determined, and the moving object can be used as the target object. Subsequently, focusing is performed based on the predicted frame and corresponding depth information of the moving object.

[0096] Figure 6 It is a depth map provided by an embodiment of the present application, in which the depth maps corresponding to the images of the nth frame, the n+1th frame, the n+2th frame and the n+3th frame are shown from left to right. The color depth in the depth map represents the depth, see Figure 5. By comparing the changes in depth information between adjacent depth maps, the currently existing moving objects can be analyzed.

[0097] The embodiments of the present application do not limit the specific method of obtaining the depth map. As an example, the depth map corresponding to the image frame can be obtained by a DTOF (Direct Time Of Flight) device. Specifically, the DTOF device is very sensitive to vertical motion in the z direction, that is, depth information. Therefore, the DTOF device can accurately detect objects moving in the vertical direction, and by determining the object as a target object and focusing based on the predicted frame and corresponding depth information of the object, it can improve the focusing performance of electronic devices for moving objects, especially moving objects in the vertical direction, and improve the shooting quality.

[0098] Specifically, after each new image frame is captured, the current moving object can be detected based on the depth map to determine the target object that needs to be focused on at the target acquisition moment. That is to say, the target object in the embodiment of the present application will change dynamically as the image frame is captured. Figure 7 This is a schematic diagram of a process for determining a target object provided by an embodiment of the present application. Figure 7 The process of determining the target object is further described.

[0099] As an example, after the sensor outputs an image, a depth map is obtained for the image frame, and the detection algorithm can predict the motion of the object in the image frame. As an example, motion prediction can be performed on one or more objects contained therein, and the prediction frame and depth information corresponding to the prediction frame of one or more objects can be obtained. In this process, the difference between consecutive depth maps is analyzed, and based on the analysis results, it is determined whether a moving object can be detected at present. If a moving object is detected, the focus frame is moved to the moving object. Specifically, the prediction frame corresponding to the detected moving object is used as the focus frame required at the target acquisition moment, and the focus is performed based on the focus frame; if no moving object is detected, the focus frame position is maintained. Specifically, the target object determined based on the depth map of the previous image frame can continue to be used as the target object, and the focus is performed based on the prediction frame and depth information corresponding to the target object.

[0100] On this basis, once a moving object is detected based on the depth map, the focus frame can be immediately moved to the predicted frame corresponding to the moving object, and focusing can be performed based on the focus frame. Therefore, once the algorithm outputs the predicted ROI, the AF scheduling can be executed, so that the in-focus position can be calculated as soon as possible and the code can be sent to the motor.

[0101] In an embodiment of the present application, the depth map corresponding to the image frame is used to detect the currently existing moving object, and the detected moving object is used as the target object. Focusing is performed based on the prediction frame and depth information corresponding to the target object. This can improve the focusing performance for the moving object, ensure that the moving object in the captured image is clear, and improve the shooting quality.

[0102] In one embodiment of the present application, the step of determining the moving object in the first image frame based on the depth information variation in the depth map and taking the moving object as the target object includes:

[0103] Based on the variation of the depth information in the depth map, the moving objects in the first image frame and the moving speed of each moving object are determined, and the moving object with the fastest moving speed is determined as the target moving object.

[0104] Specifically, by analyzing the depth information between the depth maps corresponding to the first image frame and the second image frame, the current moving object can be detected, and the moving speed of the moving object can be determined based on the magnitude of the change in the depth information, or the speed of the change in the depth information. Thus, the moving object with the fastest moving speed can be used as the target object that needs to be focused on at the target acquisition moment, and the focus can be performed based on the prediction frame and depth information corresponding to the target object.

[0105] Continuing with the previous example, the depth map corresponding to the image frame can be obtained through the DTOF device. Specifically, the DTOF device uses high-density laser to achieve depth detection, and the detection of the moving area of ​​the image frame is more accurate. Therefore, the accuracy of determining the fastest moving object is higher.

[0106] In an embodiment of the present application, the movement speed of the currently detected moving object can be analyzed based on the depth map corresponding to the captured image frame, the fastest moving object can be determined in real time, and the prediction frame corresponding to the moving object can be used as the focus frame for focusing. The focus frame can be dynamically corrected based on the depth information during the shooting process to improve the focusing performance for fast-moving objects, and ensure that when shooting dynamic scenes where the movement of the object changes in real time, each captured image frame can have good clarity, which can effectively improve the video shooting quality.

[0107] In practical applications, path delay is also one of the reasons that lead to unclear images. Figure 3 ,Under the current scheduling process, when the sensor has already output the image for the N+1th frame, the motor has not yet been pushed to the quasi-focus position, so it is impossible to ensure that the sensor outputs a clear image.

[0108] Therefore, in one embodiment of the present application, before performing motion prediction on the target object based on the first image frame including the target object and one or more second image frames including the target object captured before the first image frame, the method further includes:

[0109] The image data collected by the sensor is acquired, and the timestamps of the image data are aligned through time soft synchronization to obtain the image data corresponding to the first image frame / the second image frame.

[0110] In order to achieve clear image output from the sensor, the motor needs to complete inference and stabilize before the sensor outputs the next frame. In an embodiment of the present application, in order to stabilize the channel delay, data collected by different sensors can be synchronized and transmitted based on a soft synchronization method. Specifically, a unified host provides a reference time for each sensor, and each frame of data detected by the sensor is synchronized to a unified timestamp. As an example, time synchronization between photoelectric sensors and depth sensors can be achieved based on soft synchronization. Aligning data timestamps through time soft synchronization can facilitate subsequent calculations based on the path delay represented by the timestamp when predicting the prediction frame and depth information corresponding to the target object at the target shooting moment, so that the path delay can be compensated during the prediction process to stabilize the path delay.

[0111] In one embodiment of the present application, the step of performing motion prediction on the target object based on the first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame to obtain a predicted frame of the target object in a third image frame corresponding to the target capturing time and depth information corresponding to the predicted frame includes:

[0112] Determine the three-dimensional position coordinates of the target object in the first image frame and the second image frame based on the detection frame of the target object in the first image frame and the second image frame and the corresponding depth information, and determine the three-dimensional motion speed of the target object based on the three-dimensional position coordinates of the target object in the first image frame and the second image frame;

[0113] Based on the three-dimensional motion speed and the preset motion equation, the three-dimensional position coordinates of the target object at the target shooting time are determined, and the prediction frame of the target object in the third image frame corresponding to the target shooting time and the depth information corresponding to the prediction frame are obtained.

[0114] In an embodiment of the present application, for the first image frame and the second image frame that have been captured, the two-dimensional position coordinates (x, y) of the target object can be obtained by detecting the position of the target object therein, and combined with the depth information detected for the target object, the corresponding three-dimensional position coordinates (x, y, z) of the target object can be determined.

[0115] On the basis of determining the three-dimensional position coordinates of the target object, the speed of the target moving object in the three directions of x, y and z can be estimated based on the position parameters of the target object in the first image frame and the second image frame, and the motion equation corresponding to the target object can be determined, so that the three-dimensional coordinates of the target object at the target acquisition moment can be predicted based on the motion equation and combined with the position parameters of the target object in the first image frame, and the prediction frame of the target object in the third image frame and the depth information corresponding to the prediction frame can be obtained, wherein the depth information corresponding to the prediction frame can be specifically understood as the depth information of the feature points in the prediction frame, that is, the depth information of the target object.

[0116] The specific preset motion equation can be selected based on actual needs, and the embodiment of the present invention does not limit this. As an example, if the three-dimensional position coordinates of the target object corresponding to the N+1 frame need to be predicted, the motion equations in the x, y and z directions can be fitted based on the three-dimensional position coordinates of the target object in the previous N frames of images, so that the three-dimensional position coordinates of the target object corresponding to the N+1 frame can be calculated on this basis.

[0117] In an embodiment of the present application, the three-dimensional motion speed of the target object is determined based on the three-dimensional position coordinates corresponding to the target object in the first image frame and the second image frame, and the three-dimensional motion speed and the motion equation are combined to predict the prediction frame corresponding to the target object in the third image frame and the depth information corresponding to the prediction frame. The prediction result has good accuracy, thereby ensuring the focusing effect and improving the image capture quality.

[0118] In one embodiment of the present application, the aforementioned step of focusing based on the predicted frame and the depth information, and capturing the third image frame at the target shooting time with the focused focal length includes:

[0119] The target standard focus position is calculated based on the depth information corresponding to the predicted frame, and the lens is driven to move to the target standard focus position so that the predicted frame is focused on the imaging plane.

[0120] Specifically, focusing means changing the distance between the lens and the imaging plane so that the image of the object is clear. To achieve clear imaging, the light emitted by the object must be focused on the imaging plane corresponding to the lens, where the imaging plane is specifically the plane where the photosensitive element is located.

[0121] In the embodiment of the present application, the depth corresponding to the prediction frame is considered to be the actual depth of the target object at the target shooting moment. Therefore, based on the depth information corresponding to the prediction frame, the convergence position of the light emitted by the target object at the target shooting moment is calculated, and the quasi-focus position required to make the target object clearly imaged can be determined. By driving the lens to the quasi-focus position, focusing can be achieved.

[0122] Figure 8 is a schematic diagram of the focusing process of the shooting method provided in the embodiment of the present application. Figure 8 For further description of the embodiments of the present application, see Figure 8 After the sensor outputs the image for the Nth frame, the detection algorithm predicts the motion of the object in the image based on the Nth frame image and the image of the previous frame, predicts the N+1 frame position for the object, obtains its predicted frame in N+1, or predicted ROI and the depth information corresponding to the predicted ROI, and then the detection algorithm sends the predicted ROI. Specifically, for the channel transmission in this process, the synchronization of the timestamp specifically adopts time soft synchronization to reduce and stabilize the delay. In addition, the depth detection module analyzes the depth map corresponding to the N-1th frame and the Nth frame image, determines the currently detected moving object, moves the focus frame to the predicted ROI of the moving object, and the AF algorithm calculates the multi-window quasi-focus position. In an embodiment of the present application, focusing can be performed based on a multi-window manner, and the focusing strategy can be adjusted. Specifically, after the sensor outputs the image for the Nth frame, in addition to the ROI corresponding to the target object, multiple reference ROIs can be determined around this ROI, and the predicted ROI and depth information corresponding to the target object, as well as the predicted ROI and depth information corresponding to the reference ROI, are obtained. For each predicted ROI, the quasi-focus position is calculated, and the confidence of each quasi-focus position is calculated. Based on the quasi-focus position with a higher confidence, a code is sent to the motor. Based on the sent code, the motor pushes the lens to the specified position until it is stable. Among them, the code can be specifically used to indicate the amount of push that the motor needs to achieve.

[0123] Based on the prediction frame corresponding to the N+1th frame and the depth information obtained by the prediction, when the N+1th frame image is actually shot, although the motor position is calculated for the previous frame, the current frame is clear. After testing, using the shooting method provided by the embodiment of the present application, the driving error of the motor can be less than one focal depth.

[0124] In this process, by ensuring that the algorithm running time, the path transmission delay, and the motor stabilization time are as short as possible, it is possible to ensure that the motor is pushed to the specified position and stabilized before the sensor outputs the image for the N+1th frame, thereby avoiding the problem of out-of-focus. Therefore, in the embodiment of the present application, by analyzing the path delay, the AF scheduling process, and the effective time of the motor, combined with the motion prediction for the target object, it is possible to ensure that the sensor outputs a clear image for each frame.

[0125] Based on the same inventive concept, the embodiment of the present application provides a shooting device accordingly, such as Fig. 9 As shown, Fig. 9 A schematic diagram of a photographing device provided in an embodiment of the present invention includes:

[0126] A prediction module 901 is configured to perform motion prediction on the target object based on a first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame, to obtain a prediction frame of the target object and depth information corresponding to the prediction frame in a third image frame corresponding to a target capturing time, wherein the target capturing time is later than a capturing time of the first image frame;

[0127] The focusing module 902 is used to focus based on the predicted frame and the depth information, and capture the third image frame at the target shooting moment with the focused focal length.

[0128] The target object is motion predicted by using the first image frame and the second image frame containing the target object, and the prediction frame and the depth information corresponding to the prediction frame in the third image frame corresponding to the target shooting time are obtained, and the prediction frame and the depth information corresponding to the prediction frame are focused, and the third image frame is shot at the target shooting time with the focal length after focusing. Among them, the prediction frame and the depth information corresponding to the prediction frame obtained based on the motion prediction are closer to the real ROI and the corresponding real depth information of the target object in the third image frame than the ROI and the corresponding depth information of the target object in the current image frame that has been shot, that is, the deviation between the predicted ROI and the intended focus area is smaller. Therefore, when focusing on the third image frame before the target shooting time, the prediction frame and the depth information corresponding to the target object are used to replace the ROI and the corresponding depth information of the target object in the current image frame that has been shot by the conventional focusing algorithm. The focusing accuracy is higher and the image captured by the target object is clearer. When the target object is specifically an object in a moving state, the image blur problem caused by the moving object can be avoided, and the shooting quality can be improved.

[0129] In a specific implementation, the present application also provides a computer storage medium, wherein the computer storage medium may store a program, wherein when the program is executed, the device where the computer readable storage medium is located is controlled to perform some or all of the steps in the above embodiment. The above storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0130] In a specific implementation, an embodiment of the present application further provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes some or all of the steps in the above method embodiment.

[0131] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0132] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0133] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0134] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories (Read Only Memory, ROM), random access memories (RAM), erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0135] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0136] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0137] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0138] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A shooting method, characterized in that: include: Based on a first image frame including a target object and one or more second image frames including the target object captured before the first image frame, motion prediction is performed on the target object to obtain a predicted frame of the target object in a third image frame corresponding to a target capturing moment and depth information corresponding to the predicted frame; The target shooting time is later than the shooting time of the first image frame; Focusing is performed based on the predicted frame and the depth information, and the third image frame is captured at the target shooting moment with the focused focal length.

2. The method according to claim 1, characterized in that Determine the target object in the first image frame based on the following method: Obtaining depth maps corresponding to the first image frame and the second image frame; Based on the depth information variation in the depth map, a moving object in the first image frame is determined, and the moving object is used as the target object.

3. The method according to claim 2, characterized in that The step of determining a moving object in the first image frame based on the depth information variation in the depth map and taking the moving object as the target object comprises: Based on the depth information variation in the depth map, the moving objects in the first image frame and the moving speed of each of the moving objects are determined, and the moving object with the fastest moving speed is determined as the target moving object.

4. The method according to claim 1, characterized in that Before performing motion prediction on the target object based on the first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame, the method further includes: Image data collected by the sensor is acquired, and timestamps of the image data are aligned through time soft synchronization to obtain image data corresponding to the first image frame / the second image frame.

5. The method according to claim 1, characterized in that The step of performing motion prediction on the target object based on a first image frame including the target object and one or more second image frames including the target object captured before the first image frame to obtain a predicted frame of the target object in a third image frame corresponding to the target capturing moment and depth information corresponding to the predicted frame comprises: Determine the three-dimensional position coordinates of the target object in the first image frame and the second image frame based on the detection frames of the target object in the first image frame and the second image frame and the corresponding depth information, and determine the three-dimensional motion speed of the target object based on the three-dimensional position coordinates of the target object in the first image frame and the second image frame; Based on the three-dimensional motion speed and the preset motion equation, the three-dimensional position coordinates of the target object at the target shooting moment are determined, and the prediction frame of the target object in the third image frame corresponding to the target shooting moment and the depth information corresponding to the prediction frame are obtained.

6. The method according to claim 1, characterized in that The step of focusing based on the prediction frame and the depth information, and capturing the third image frame at the target shooting moment with the focused focal length includes: A target standard focus position is calculated based on the depth information corresponding to the prediction frame, and the lens is driven to move to the target standard focus position so that the prediction frame is focused on the imaging plane.

7. A photographing device, characterized in that: include: A prediction module, configured to perform motion prediction on the target object based on a first image frame containing the target object and one or more second image frames containing the target object captured before the first image frame, to obtain a prediction frame of the target object in a third image frame corresponding to a target capturing moment and depth information corresponding to the prediction frame; The target shooting time is later than the shooting time of the first image frame; A focusing module is used to focus based on the prediction frame and the depth information, and to capture the third image frame at the target shooting moment with a focused focal length.

8. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images