Interpolation processing method, display device, medium and program product

Abstract

The present application discloses a frame insertion processing method, display device, medium and program product. The method includes: taking the image block to be inserted as a reference, determining the data range of the first data to be written; if the data amount of the first data to be written within the data range is higher than the storage capacity of the designated storage area, downsampling the first data to be written to obtain the second data to be written; storing the second data to be written in the designated storage area; searching the first search image block in the first reference frame and the second search image block in the second reference frame from the designated storage area based on the motion vector of the image block to be inserted; generating the image content of the image block to be inserted based on the first search image block and the second search image block. In this way, the motion compensation search range in the frame insertion process can be increased and the loss of image quality can be reduced by downsampling the data to be written.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a frame insertion processing method, a display device, a medium and a program product. Background Art

[0002] Motion estimation and motion compensation is a technology currently widely used in frame rate conversion. It estimates the motion trajectory of objects in continuous moving images, combines image data and the obtained motion vector, and interpolates intermediate image frames, thereby improving the image frame rate and improving problems such as jitter and tailing during video playback.

[0003] In the prior art, a large amount of data needs to be processed during the interpolation process, and sometimes the quality of the interpolated frames obtained by motion estimation and motion compensation still needs to be improved. Summary of the invention

[0004] The purpose of the present application is to provide a frame insertion processing method, a display device, a medium and a program product, so as to solve the problem that the image quality obtained by motion compensation search during the frame insertion process still needs to be improved.

[0005] In a first aspect, the present application provides a frame interpolation processing method, wherein a first reference frame and a second reference frame are used to synthesize an image frame to be synthesized, wherein the image frame to be synthesized includes a plurality of image blocks to be interpolated, and each image block to be interpolated has a corresponding motion vector. The method includes:

[0006] Determine the data range of the first data to be written based on the image block to be inserted;

[0007] If the amount of the first data to be written in the data range is greater than the storage capacity of the designated storage area, downsampling the first data to be written to obtain the second data to be written;

[0008] storing the second to-be-written data in the designated storage area;

[0009] Searching the designated storage area for a first search image block in the first reference frame and a second search image block in the second reference frame based on a motion vector of the image block to be interpolated;

[0010] The image content of the image block to be inserted is generated based on the first search image block and the second search image block.

[0011] In a possible implementation manner, determining the data range of the first data to be written based on the image block to be inserted into the frame specifically includes:

[0012] For the image block group where the image block to be interpolated is located, determine the center position of a specified direction within the image block group, wherein if the image block group is a row of image blocks, the specified direction is a pixel row, and if the image block group is a column of image blocks, the specified direction is a pixel column;

[0013] The data range of the first data to be written is determined based on the center position of the specified direction, and the data range includes pixel points in the first specified range and pixel points in the second specified range; wherein the distance between the pixel points in the second specified range and the center position is greater than the distance between the pixel points in the first specified range and the center position.

[0014] In a possible implementation manner, downsampling the first data to be written to obtain the second data to be written specifically includes:

[0015] Down-sampled data are obtained after down-sampling processing is performed on the pixel points in the second specified range in the first reference frame and the second reference frame respectively, and the down-sampled data and the pixel points in the first specified range are used as the second data to be written.

[0016] In a possible implementation manner, the downsampling of the pixels in the second specified range in the first reference frame and the second reference frame respectively specifically includes:

[0017] Execute the following steps for the pixel points in the second specified range in the first reference frame and the second reference frame respectively:

[0018] If the designated direction is a pixel row, for any adjacent multiple rows of pixel points within the second designated range, the number of pixel rows is reduced by taking an average or weighted sum;

[0019] If the designated direction is a pixel column, for any adjacent multiple columns of pixels within the second designated range, the number of pixel columns is reduced by taking an average or a weighted sum;

[0020] If the specified direction includes pixel rows and pixel columns, then for any adjacent multiple rows of pixel points within the second specified range, the pixel rows are reduced by averaging or weighted summing, and the number of pixel columns is reduced by averaging or weighted summing for multiple columns of pixel points after the pixel rows are reduced; or, if the specified direction includes pixel rows and pixel columns, then for any adjacent multiple columns of pixel points within the second specified range, the pixel columns are reduced by averaging or weighted summing, and the number of pixel rows is reduced by averaging or weighted summing for multiple rows of pixel points after the pixel columns are reduced.

[0021] In a possible implementation manner, the downsampling of the pixels in the second specified range in the first reference frame and the second reference frame respectively specifically includes:

[0022] Determining a motion vector level for each of the first reference frame and the second reference frame based on the motion vector of each to-be-interpolated image block in the image block group;

[0023] Based on the downsampling rate corresponding to the motion vector level, downsampling processing is performed on the pixel points in the second specified range of the first reference frame and the second reference frame respectively.

[0024] In a possible implementation manner, searching the designated storage area for a first search image block in the first reference frame and a second search image block in the second reference frame based on the motion vector of the image block to be interpolated specifically includes:

[0025] Based on the motion vector of the image block to be interpolated and the center position of the image block to be interpolated in a specified direction, acquiring the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction;

[0026] Based on the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction, data of the first search image block in the first reference frame and data of the second search image block in the second reference frame are read from the specified storage area.

[0027] In a possible implementation manner, reading data of the first search image block in the first reference frame and data of the second search image block in the second reference frame from the designated storage area based on the center position of the first search image block in the designated direction and the center position of the second search image block in the designated direction specifically includes:

[0028] When the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction are within the first specified range, reading data of the first search image block and the second search image block in the specified storage area;

[0029] When the center position of the specified direction of the first search image block and the center position of the specified direction of the second search image block are within the second specified range, the area range of the first search image block and the second search image block are determined based on the downsampling processing method; data is read from the specified storage area based on the area range of the first search image block and the area range of the second search image block, and an upsampling operation is performed on the read data to restore the data amount of the first search image block and the data amount of the second search image block, so that the data amount of the first search image block and the data amount of the second search image block are the same as the data amount of the image block to be interpolated.

[0030] In a second aspect, the present application provides a display device, including:

[0031] Display, processor and memory;

[0032] The display is used to display the image;

[0033] The memory is used to store the processor executable instructions;

[0034] The processor is configured to execute the instructions to implement the frame insertion processing method as described in any one of the first aspects above.

[0035] In a third aspect, the present application provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a display device, the display device can perform the interpolation processing method as described in any one of the first aspects above.

[0036] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the interpolation processing method as described in any one of the first aspects above.

[0037] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0038] The embodiment of the present application provides a frame insertion processing method, a display device, a medium and a program product, wherein a first reference frame and a second reference frame are used to synthesize an image frame to be synthesized, wherein the image frame to be synthesized includes a plurality of image blocks to be inserted, each of which has a corresponding motion vector, and the data range of the first data to be written is determined by taking the image block to be inserted as a reference; if the data amount of the first data to be written within the data range is higher than the storage capacity of the designated storage area, the first data to be written is downsampled to obtain the second data to be written; the second data to be written is stored in the designated storage area; based on the motion vector of the image block to be inserted, the first search image block in the first reference frame and the second search image block in the second reference frame are searched from the designated storage area; and the image content of the image block to be inserted is generated based on the first search image block and the second search image block. Thus, the frame insertion operation can be completed by downsampling the data to be written and compressing the data to be stored in a storage area with limited capacity, and data can be dynamically read according to the motion vector corresponding to the image block to be inserted, thereby increasing the search range of motion compensation during the frame insertion process, so as to ensure that motion compensation obtains more useful information, thereby reducing the loss of image quality.

[0039] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of the movement of the image block to be inserted in the y-axis direction of the coordinate axis provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a motion compensation search range provided in an embodiment of the present application;

[0043] Figure 3 A usage scenario of the display device provided in the embodiment of the present application;

[0044] Figure 4 A hardware configuration block diagram of the control device 100 provided in an embodiment of the present application;

[0045] Figure 5 A hardware configuration block diagram of a display device 200 provided in an embodiment of the present application;

[0046] Figure 6 A software configuration diagram of the display device 200 provided in an embodiment of the present application;

[0047] Figure 7 A flowchart of a frame insertion processing method provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of a motion speed range of an image block to be inserted into a frame provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of data storage provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] Furthermore, in the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in the text is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “plurality” refers to two or more than two.

[0052] In the following, the terms "first", "second", and "second" are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0053] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0054] 1) Motion Estimation and Motion Compensation (MEMC): It is a technology widely used in frame rate conversion. It estimates the motion trajectory of objects in continuous motion images, combines image data and the obtained motion vector, and interpolates intermediate images to improve image frame rate and improve jitter and tailing problems during video playback. Motion Estimation and Motion Compensation is divided into two parts: Motion Estimation (ME) and Motion Compensation (MC).

[0055] 2) Motion Estimation (ME): The object motion vector MV is calculated through an algorithm. The object motion vector MV represents the size of the object's motion displacement.

[0056] 3) Motion Compensation (MC): Generate new image frames based on MV.

[0057] 4) SRAM (Static Random-Access Memory) chip: It is a high-speed cache placed between the CPU and the main memory to speed up the transmission of data within the CPU. SRAM is fast and does not need to be refreshed, but it is expensive and large in size, so it cannot be used as a large-scale main memory.

[0058] The execution process of motion compensation is introduced below to help those skilled in the art further understand the technical solution provided in the embodiments of the present application.

[0059] In the process of MEMC processing, the image frame to be inserted is processed in units of image blocks. For example, the image frame to be inserted with a resolution of 4K (3840x2160) can be divided into 960x540 image blocks of 4x4. The data of the image block in the previous frame and the data of the image block in the next frame are obtained according to the MV of the image block in the frame to be inserted, and then the data of the image block in the previous frame and the data of the image block in the next frame are obtained according to the corresponding mixing algorithm. When all the image blocks in the frame to be inserted are processed, a new image frame is generated.

[0060] Each image block corresponds to a motion vector MV calculated in the ME process, and the motion vector MV of each image block is composed of displacements in two directions of the x-axis and the y-axis.

[0061] Reference Figure 1 , is a schematic diagram of the movement of the image block to be inserted in the y-axis direction of the coordinate axis provided in an embodiment of the present application. Figure 1As shown, I and P represent I frame and P frame respectively, which are the previous frame and the next frame of the image frame to be inserted. The vertical dotted line represents the position of the image frame to be inserted, and the position where the two horizontal and vertical dotted lines intersect is the current processed image block of the image frame to be inserted. The position pointed to by the solid line 1 indicates that the current image block to be inserted has no motion, and its relative position in the I and P frames is fixed, while the position pointed to by the solid line 2 indicates that the current image block to be inserted moves from the top in the I frame to the bottom in the P frame, with a large motion displacement. When the image block to be inserted has a large motion displacement, it is necessary to search for data in a larger range. Since the data capacity of the storage area is limited, the data to be searched may exceed the data range stored in the storage area. Therefore, the embodiment of the present application provides a solution that can expand the search range and search for as much required data as possible to improve the image quality of the interpolated image.

[0062] like Figure 2 , which is a schematic diagram of the motion compensation search range provided in an embodiment of the present application. Figure 2 The bold lines 1 and 2 shown represent the data stored in the storage area. The more data stored, the larger the range of values ​​that can be taken at the location of the image block to be interpolated, and the larger the motion search range. However, if too little data is stored, when the value range exceeds the data range of the storage area, the required data cannot be obtained, which may cause the image quality of the image to be interpolated to be lost, or even cause part of the picture to be missing. For example, SRAM is used to store the data of the previous frame image and the next frame image, but the space resources of SRAM are limited and the entire row of data is stored. When the image block to be interpolated moves left and right, no matter how large the motion displacement is, the corresponding data can always be obtained in SRAM. When the image block to be interpolated moves up and down, the motion compensation search range is limited to the number of rows stored in SRAM data. The more rows stored, the larger the search range; the fewer rows stored, the smaller the search range. When the value range exceeds the data rows stored in SRAM, the required data cannot be obtained, resulting in data loss.

[0063] In the related art, a large amount of data needs to be processed during the interpolation process, but the motion compensation search range during the interpolation process is limited, which may result in incomplete access to data and increase the loss of image quality.

[0064] In view of this, the present application proposes a frame insertion processing method, a display device, a medium and a program product to solve the problem that the image quality obtained by motion compensation search during the frame insertion process still needs to be improved.

[0065] The inventive concept of the present application can be summarized as follows: in the present application, the first reference frame and the second reference frame are used to synthesize the image frame to be synthesized, the image frame to be synthesized includes a plurality of image blocks to be interpolated, each image block to be interpolated has a corresponding motion vector, and the data range of the first data to be written is determined by taking the image block to be interpolated as a reference; if the data amount of the first data to be written within the data range is higher than the storage capacity of the designated storage area, the first data to be written is downsampled to obtain the second data to be written; the second data to be written is stored in the designated storage area; based on the motion vector of the image block to be interpolated, the first search image block in the first reference frame and the second search image block in the second reference frame are searched from the designated storage area; the image content of the image block to be interpolated is generated based on the first search image block and the second search image block. Thus, the interpolation operation can be completed by downsampling the data to be written, compressing the data and storing it in a storage area with limited capacity, and at the same time, the data can be dynamically read according to the motion vector corresponding to the image block to be interpolated, thereby increasing the search range of motion compensation in the interpolation process to ensure that motion compensation obtains more useful information, thereby reducing the loss of image quality.

[0066] After introducing the inventive concept of the present application, the usage scenario of the display device provided by the present application is described below. Figure 3 Schematic diagram of a usage scenario of a display device provided in an embodiment of the present application. Figure 3 As shown, the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0067] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and at least one of other short-range communication methods, to control the display device 200 wirelessly or wired. The user may control the display device 200 by inputting user commands through at least one of buttons on the remote controller, voice input, and control panel input.

[0068] In some embodiments, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a laptop computer, an AR / VR device, etc.

[0069] In some embodiments, the smart device 300 may also be used to control the display device 200. For example, the display device 200 may be controlled using an application running on the smart device.

[0070] In some embodiments, the smart device 300 and the display device may also be used to communicate data.

[0071] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0072] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0073] In some embodiments, the software steps executed by a step execution subject can be migrated to another step execution subject in data communication with it as needed for execution. Exemplarily, the software steps executed by a server can be migrated to a display device in data communication with it as needed for execution, and vice versa.

[0074] Figure 4 This is a block diagram of the configuration of the control device 100 provided in the embodiment of the present application. Figure 4 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive the user's input operation instructions and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play the role of an interactive intermediary between the user and the display device 200.

[0075] In some embodiments, the communication interface 130 is used for communicating with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module.

[0076] In some embodiments, the user input / output interface 140 includes at least one of a microphone, a touch pad, a sensor, a button, or alternative modules.

[0077] The following is a detailed description of the embodiment using the display device 200 as an example. It should be understood that: Figure 5 The display device 200 shown is only an example, and the display device 200 may have more Figure 5 The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0078] Figure 5 A hardware configuration block diagram of the display device 200 provided in an embodiment of the present application.

[0079] In some embodiments, the display apparatus 200 includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0080] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output.

[0081] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface, etc.

[0082] In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

[0083] In some embodiments, the tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0084] In some embodiments, the communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0085] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0086] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following interfaces: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0087] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0088] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0089] In some embodiments, the object may be any one of the selectable objects, such as a hyperlink, an icon or other operable controls. Operations related to the selected object include: displaying a page, document, image, etc. connected to the hyperlink, or executing a program corresponding to the icon.

[0090] In some embodiments, the controller includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0091] CPU processor. Used to execute operating system and application instructions stored in the memory, and to execute various applications, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video content. CPU processor may include multiple processors. For example, it includes a main processor and one or more sub-processors.

[0092] In some embodiments, the graphics processor is used to generate various graphic objects, such as at least one of icons, operation menus, and user input command display graphics. The graphics processor includes an operator, which performs operations by receiving various interactive instructions input by the user and displays various objects according to display attributes; and a renderer, which renders various objects obtained based on the operator, and the rendered objects are used to be displayed on the display.

[0093] In some embodiments, the video processor is used to receive an external video signal and perform at least one of the following video processing operations, such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis, according to a standard codec protocol of the input signal, to obtain a signal that can be directly displayed or played on the display device 200.

[0094] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, and a display formatting module. The demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to the user input or itself with the scaled video image to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to receive the video output signal after frame rate conversion and change the signal to conform to the display format signal, such as outputting an RGB data signal.

[0095] In some embodiments, the audio processor is used to receive an external audio signal, perform decompression and decoding according to a standard codec protocol of the input signal, and perform at least one of noise reduction, digital-to-analog conversion, and amplification processing to obtain a sound signal that can be played in a speaker.

[0096] In some embodiments, the user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0097] In some embodiments, "user interface" refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on a display screen of an electronic device, wherein a control may include at least one of icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0098] In some embodiments, the user interface 280 is an interface that can be used to receive control input (such as a physical button on the display device body, or other interfaces).

[0099] In some embodiments, the system of the display device may include a kernel, a command parser (shell), a file system, and an application. The kernel, shell, and file system together constitute the basic operating system structure, which allows users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, schedulers, signals, and inter-process communication (IPC). After the kernel starts, the shell and user applications are loaded. After startup, the application is compiled into machine code to form a process.

[0100] See also Figure 6 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Appiication Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0101] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0102] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0103] like Figure 6 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0104] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0105] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0106] In some embodiments, the kernel layer is a layer between hardware and software. Figure 6 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0107] To further illustrate the technical solution provided by the embodiments of the present application, a detailed description is given below in conjunction with the accompanying drawings and specific implementation methods.

[0108] refer to Figure 7 A flowchart of a frame insertion processing method provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, in the method, the first reference frame and the second reference frame are used to synthesize the image frame to be synthesized, and the image frame to be synthesized includes a plurality of image blocks to be interpolated, each of which has a corresponding motion vector. The method includes the following steps:

[0109] In step 701, the data range of the first data to be written is determined based on the image block to be inserted.

[0110] In a possible implementation, in order to increase the processing efficiency of determining the data range of the first data to be written, an image block group is introduced in the present application, a row of image blocks is an image block group or a column of image blocks is an image block group, and then the data range for interpolation processing is obtained in units of image block groups. It can be implemented as determining the data range of the first data to be written based on the image block to be interpolated, and the center position of a specified direction within the image block group where the image block to be interpolated is located can be determined, and the data range of the first data to be written is determined based on the center position of the specified direction.

[0111] If the image block group is a row of image blocks, the specified direction is the pixel row; if the image block group is a column of image blocks, the specified direction is the pixel column. The data range includes the pixel points of the first specified range and the pixel points of the second specified range. The distance between the pixel points of the second specified range and the center position is greater than the distance between the pixel points of the first specified range and the center position. Exemplarily, assuming that a row of image blocks is an image block group, the middle position of the row where the image blocks to be interpolated are located with a size of 4x4 can be defined as the center position, that is, the bottom of the second row and the top of the third row of the image blocks to be interpolated are used as the center position of the image block group, and then the data range of the first data to be written is determined based on this center. For example, 50 rows of data can be taken from the top and bottom as the data to be written in the first specified range, and then the data of 51-220 rows above and below can be taken as the data to be written in the second specified range for downsampling and storage in the specified storage area (such as SRAM).

[0112] Among them, when processing image blocks in the same image block group, it is not necessary to update the data range of the first data to be written. When all image blocks in the same image block group are processed, the data in the designated storage area is updated, that is, another row of image blocks is regarded as an image block group, and the center position is determined based on the new image block group, and the data range of the first data to be written is determined and written into the designated storage area.

[0113] Therefore, a row of image blocks can be directly regarded as an image block group, and the data of a row of image blocks can be processed simultaneously each time, thereby speeding up the processing efficiency of determining the data range of the first data to be written.

[0114] In step 702, if the data amount of the first data to be written in the data range is greater than the storage capacity of the designated storage area, downsampling processing is performed on the first data to be written to obtain second data to be written.

[0115] In a possible implementation, if the object is stationary or moving at a low speed, its details are easy to observe, and when the object is moving at a high speed, its details will be difficult to observe. Therefore, the data of the image blocks to be interpolated that are stationary or moving at a low speed can be directly stored in a designated storage area, and the data of the image blocks to be interpolated that are moving at a high speed are downsampled to ensure a larger search range for the image blocks to be interpolated that are moving at a high speed, thereby minimizing the loss of image quality.

[0116] Therefore, in the present application, the first data to be written is downsampled to obtain second data to be written, including downsampling the pixels in the second specified range in the first reference frame and the second reference frame respectively to obtain downsampled data, and the downsampled data and the pixels in the first specified range are used as the second data to be written, that is, the data in the second specified range is the data of the image block with a larger motion speed.

[0117] Among them, the pixel points in the first specified range in the first data to be written are the original data of the image block and do not need to be downsampled, and the pixel points in the second specified range are the data that need to be downsampled and are used to store the processed data.

[0118] For example, Figure 8 As shown, it is a schematic diagram of the range of motion speed of the image block to be interpolated provided in an embodiment of the present application. The horizontal dotted line indicates that the image block to be interpolated is in the same position as the image block of the previous frame and the image block of the next frame, that is, the image block to be interpolated is stationary, and the two oblique lines represent the two directions of movement, upward and downward, respectively. The greater the slope of the oblique line, the greater the motion vector of the image block to be interpolated, and the greater the motion speed of the image block to be interpolated when the time between the two frames is fixed. When the data in the first specified range is obtained, it means that the motion speed of the image block to be interpolated is relatively small, and when the data in the second specified range is obtained, it means that the motion speed of the image block to be interpolated is relatively large. If the data of the image in this area is downsampled, it can be ensured that the image blocks to be interpolated with a large motion speed are downsampled, thereby increasing the motion compensation search range.

[0119] The motion speed of the image block to be inserted can be represented by a motion vector. When the time between two frames is fixed, the larger the motion vector, the greater the motion speed, and the smaller the motion vector, the smaller the motion speed.

[0120] Thus, downsampling is performed on the data in the second designated range, ensuring that more data can be stored in the designated storage area, and increasing the motion compensation search range of the image blocks to be inserted with a relatively high motion speed.

[0121] In a possible implementation manner, the present application performs downsampling processing on the pixels in the second specified range in the first reference frame and the second reference frame respectively, and may perform the following on the pixels in the second specified range in the first reference frame and the second reference frame respectively:

[0122] If the designated direction is a pixel row, for any adjacent multiple rows of pixel points within the second designated range, the number of pixel rows is reduced by taking an average or weighted sum;

[0123] If the designated direction is a pixel column, for any adjacent multiple columns of pixels within the second designated range, the number of pixel columns is reduced by taking an average or a weighted sum;

[0124] If the specified direction includes pixel rows and pixel columns, then for any adjacent multiple rows of pixel points within the second specified range, the pixel rows are reduced by averaging or weighted summing, and the number of pixel columns is reduced by averaging or weighted summing for multiple columns of pixel points after the pixel rows are reduced; or, if the specified direction includes pixel rows and pixel columns, then for any adjacent multiple columns of pixel points within the second specified range, the pixel columns are reduced by averaging or weighted summing, and the number of pixel rows is reduced by averaging or weighted summing for multiple rows of pixel points after the pixel columns are reduced.

[0125] For example, Fig. 9 As shown, it is a schematic diagram of storing data provided by an embodiment of the present application. Fig. 9 160 rows of data can be stored in the image, of which the middle 100 rows retain the original data, and the 30 rows on each side store the downsampled data. For example, a 2x downsampling rate can be used to add the pixel values ​​of the corresponding two pixels of any two adjacent rows of data on each side to obtain the average value and synthesize one row of data. In this way, the 30 rows of data on each side actually become 60 rows of data on each side, and the search range is increased from the original 160 (30+100+30) rows of data to 220 (30x2+100+30x2) rows of data.

[0126] The downsampling rate can be fixed at 2 times the downsampling rate, or the register can be increased to flexibly configure multiples. For example, a portion of row data can use a 2 times downsampling rate, and a portion of higher speed data rows can use a 3 times downsampling rate.

[0127] For example, Fig. 9 30 rows on each side are used to store the downsampled data. Assuming that the image block to be inserted is 4x4 in size, the weight of the first row of data can be set to 20%, the weight of the second row of data to 40%, the weight of the third row of data to 30%, and the weight of the fourth row of data to 10%. In this way, the pixel values ​​of the four pixels corresponding to any four adjacent rows of data are weighted and summed according to the corresponding weights to synthesize one row of data. In this way, 30 rows of data on each side actually become 120 rows of data stored on each side, and the search range is increased from the original 160 (30+100+30) rows of data to 340 (30x4+100+30x4) rows of data. Of course, the specific downsampling method can be flexibly set and is applicable to the embodiments of the present application.

[0128] In another possible implementation, the present application can also flexibly determine the downsampling rate. For example, downsampling is performed on the pixels in the second specified range in the first reference frame and the second reference frame respectively, and the motion vector level can be determined for the first reference frame and the second reference frame respectively based on the motion vectors of each image block to be interpolated in the image block group; based on the downsampling rate corresponding to the motion vector level, downsampling is performed on the pixels in the second specified range in the first reference frame and the second reference frame respectively.

[0129] For example, Fig. 9 30 lines on each side of the middle store the data after downsampling processing. The dotted line can be set as the center. If the distance is more than 50 lines and less than 60 lines from the center, the motion vector level is determined to be the first level, and a 2-fold downsampling rate is used for downsampling processing. If the distance is more than 60 lines and less than 70 lines from the center, the motion vector level is determined to be the second level, and a 3-fold downsampling rate is used for downsampling processing. If the distance is more than 70 lines and less than 80 lines from the center, the motion vector level is determined to be the third level, and a weighted summation method is used for downsampling processing. In this way, different downsampling rates can be used for downsampling processing according to different motion vector levels, and the quality of the picture can be better guaranteed according to different motion vector levels. That is, the lower the motion vector level, the better the picture quality can be guaranteed.

[0130] Therefore, the pixels in the second specified range can be downsampled by averaging, weighted summing, using different downsampling rates according to different motion vector levels, etc., appropriately sacrificing image quality so that more data can be stored in the same storage space.

[0131] In step 703, the second data to be written is stored in the designated storage area.

[0132] In step 704, a first search image block in the first reference frame and a second search image block in the second reference frame are searched from the designated storage area based on the motion vector of the image block to be interpolated.

[0133] In a possible implementation, before reading the data for synthesizing the image block to be interpolated from the first reference frame and the second reference frame, it is also necessary to determine the first search image block and the second search image block corresponding to the image block to be interpolated in the first reference frame and the second reference frame. Therefore, in the present application, the first search image block in the first reference frame and the second search image block in the second reference frame are searched from the designated storage area based on the motion vector of the image block to be interpolated, and the center position of the first search image block in the designated direction and the center position of the second search image block in the designated direction can be obtained based on the motion vector of the image block to be interpolated and the center position of the image block to be interpolated in the designated direction; and based on the center position of the first search image block in the designated direction and the center position of the second search image block in the designated direction, the data of the first search image block in the first reference frame and the data of the second search image block in the second reference frame are read from the designated storage area.

[0134] Exemplarily, during the interpolation process, a frame image is divided into 960x540 image blocks of size 4x4. The coordinates of the upper left corner position of the frame image are [0, 0], the coordinates of the upper right corner position are [959, 0], the coordinates of the lower left corner position are [0, 539], and the coordinates of the lower right corner position are [959, 539]. Assuming that the center position of the current image block to be interpolated is [90, 90] and the motion vector is (4, 4), the center position of the first search image block in the first reference frame is obtained as [86, 86], and the center position of the second search image block in the second reference frame is obtained as [94, 94]. Therefore, the data of the first search image block and the second search image block in the second reference frame can be read according to the center position of the first search image block in the first reference frame [86, 86] and the center position of the second search image block in the second reference frame [94, 94].

[0135] In this way, the center positions of the first search image block and the second search image block corresponding to the image block to be interpolated in the first reference frame and the second reference frame can be determined, so as to obtain data of the first search image block and the second search image block corresponding to the image block to be interpolated in the first reference frame and the second reference frame according to the center positions.

[0136] In a possible implementation, after determining the center positions of the first search image block and the second search image block corresponding to the image block to be interpolated in the first reference frame and the second reference frame, data of the first search image block and the second search image block are obtained according to the center positions. However, since the data of the first specified range and the second specified range are processed differently when stored, in order to accurately read the required data from the specified storage area, in the present application, based on the center position of the specified direction of the first search image block and the center position of the specified direction of the second search image block, the data of the first search image block in the first reference frame and the data of the second search image block in the second reference frame are read from the specified storage area, which can be implemented as follows:

[0137] If the required data has not been downsampled, when the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction are within the first specified range, the data of the first search image block and the second search image block in the specified storage area are read.

[0138] If the required data has been downsampled, when the center position of the specified direction of the first search image block and the center position of the specified direction of the second search image block are within the second specified range, the area range of the first search image block and the second search image block are determined based on the downsampling processing method; data is read from the specified storage area based on the area range of the first search image block and the area range of the second search image block, and the read data is upsampled to restore the data amount of the first search image block and the data amount of the second search image block, so that the data amount of the first search image block and the data amount of the second search image block are the same as the data amount of the image block to be interpolated.

[0139] For example, Fig. 9 As shown, the middle 100 rows retain the original data, and the 30 rows on each side store the data after downsampling. When reading the data of the first search image block in the first reference frame and the data of the second search image block in the second reference frame from the designated storage area, if the center position of the designated direction of the first search image block and the center position of the designated direction of the second search image block are within 50 rows above and below the dotted line position of the I and P frame image blocks, since the data stored in these 100 rows are original data and have not been downsampled, the number of rows can correspond one to one, so the data of the corresponding number of rows can be directly read for use.

[0140] If the center position of the specified direction of the first search image block and the center position of the specified direction of the second search image block exceed 50 lines above and below the dotted line position of the I and P frame image blocks, and are less than or equal to 80 lines above and below, it is necessary to determine the downsampling processing method used when storing the data. If the method of averaging the 2x downsampling rate is used for downsampling processing, the row Y close to the center position is used to determine the new starting position using the following formula, and 2 lines of data are taken outward according to the new starting position, and copied into 4 lines of data to obtain the corresponding data for use, where ceil means rounding up:

[0141] 50+ceil((Y-50)) / 2 (1)

[0142] For example, each image block of size 4x4 needs to take 4 lines of data, and after calculation, the new starting position is 51. If it has not been downsampled, then four lines of data 51, 52, 53, and 54 need to be taken. However, after downsampling at a 2x downsampling rate, the actual stored data becomes two lines of data 51 (original 51, 52) and 52 (original 53, 54). Therefore, it is necessary to take 2 lines of data out according to the new starting position 51 to obtain two lines of data 51 and 52, and copy one line of each of the two lines of data, and finally get 4 lines of data to use.

[0143] Among them, the method for determining the new starting position can also be determined using other methods according to actual conditions, and this application does not impose any restrictions on this.

[0144] If a 3-fold downsampling rate or a weighted summation method is used for downsampling, corresponding data is obtained and used according to the corresponding upsampling method.

[0145] If different downsampling methods are used for downsampling processing according to the different motion vector levels of each image block to be interpolated in the image block group, it is necessary to first determine the area range of the first search image block and the second search image block, so as to determine the motion vector level of each image block to be interpolated, and then upsampling processing is performed according to the downsampling method corresponding to the motion vector level to obtain corresponding data.

[0146] Therefore, it is possible to determine whether the center positions of the first search image block and the second search image block are within the first specified range or the second specified range, and then perform corresponding data processing when reading the data based on the data processing method used when storing the data, so as to accurately read the required data from the specified storage area and restore the original data volume for use in generating the image content of the image block to be interpolated.

[0147] In step 705, the image content of the image block to be inserted is generated based on the first search image block and the second search image block.

[0148] In a possible implementation, the image content of the image block to be interpolated may be generated according to a Blend algorithm based on the acquired data content of the first search image block and the second search image block, or may be generated according to other methods, which is not limited in this embodiment of the present application.

[0149] Thus, the image content of the image block to be inserted is generated based on the first reference frame and the second reference frame, and all the image blocks to be inserted are processed, that is, all the content of a frame to be synthesized is synthesized.

[0150] Based on the foregoing description, the embodiment of the present application determines the data range of the first data to be written by taking the image block to be interpolated as a reference; if the data amount of the first data to be written within the data range is higher than the storage capacity of the designated storage area, the first data to be written is downsampled to obtain the second data to be written; the second data to be written is stored in the designated storage area; based on the motion vector of the image block to be interpolated, the first search image block in the first reference frame and the second search image block in the second reference frame are searched from the designated storage area; the image content of the image block to be interpolated is generated based on the first search image block and the second search image block. The interpolation operation can be completed by downsampling the data to be written and compressing the data to store it in a storage area with limited capacity. At the same time, data can be dynamically read according to the motion vector corresponding to the image block to be interpolated, thereby increasing the motion compensation search range during the interpolation process to ensure that motion compensation obtains more useful information, thereby reducing the loss of image quality.

[0151] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A frame insertion processing method, characterized in that: The first reference frame and the second reference frame are used to synthesize an image frame to be synthesized, wherein the image frame to be synthesized includes a plurality of image blocks to be interpolated, each of which has a corresponding motion vector. The method includes: Taking the image block to be interpolated as a reference, for the image block group where the image block to be interpolated is located, determining the center position of a specified direction within the image block group; determining a data range of first data to be written based on the center position of the specified direction, the data range including pixel points in a first specified range and pixel points in a second specified range; wherein the distance between the pixel points in the second specified range and the center position is greater than the distance between the pixel points in the first specified range and the center position; If the amount of the first data to be written in the data range is greater than the storage capacity of the designated storage area, downsampling the pixels in the second designated range in the first reference frame and the second reference frame to obtain downsampled data, and using the downsampled data and the pixels in the first designated range as the second data to be written; storing the second to-be-written data in the designated storage area; Searching the designated storage area for a first search image block in the first reference frame and a second search image block in the second reference frame based on a motion vector of the image block to be interpolated; The image content of the image block to be inserted is generated based on the first search image block and the second search image block.

2. The method according to claim 1, characterized in that If the image block group is a row of image blocks, the specified direction is a pixel row; if the image block group is a column of image blocks, the specified direction is a pixel column.

3. The method according to claim 1, characterized in that The downsampling of the pixels in the second specified range in the first reference frame and the second reference frame respectively specifically includes: Execute the following steps for the pixel points in the second specified range in the first reference frame and the second reference frame respectively: If the designated direction is a pixel row, for any adjacent multiple rows of pixel points within the second designated range, the number of pixel rows is reduced by taking an average or weighted sum; If the designated direction is a pixel column, for any adjacent multiple columns of pixels within the second designated range, the number of pixel columns is reduced by taking an average or a weighted sum; If the specified direction includes pixel rows and pixel columns, then for any adjacent multiple rows of pixel points within the second specified range, the pixel rows are reduced by averaging or weighted summing, and the number of pixel columns is reduced by averaging or weighted summing for multiple columns of pixel points after the pixel rows are reduced; or, if the specified direction includes pixel rows and pixel columns, then for any adjacent multiple columns of pixel points within the second specified range, the pixel columns are reduced by averaging or weighted summing, and the number of pixel rows is reduced by averaging or weighted summing for multiple rows of pixel points after the pixel columns are reduced.

4. The method according to claim 1, characterized in that: The downsampling of the pixels in the second specified range in the first reference frame and the second reference frame respectively specifically includes: Determining a motion vector level for each of the first reference frame and the second reference frame based on the motion vector of each to-be-interpolated image block in the image block group; Based on the downsampling rate corresponding to the motion vector level, downsampling processing is performed on the pixel points in the second specified range of the first reference frame and the second reference frame respectively.

5. The method according to claim 1, characterized in that The step of searching the designated storage area for a first search image block in the first reference frame and a second search image block in the second reference frame based on the motion vector of the image block to be interpolated specifically includes: Based on the motion vector of the image block to be interpolated and the center position of the image block to be interpolated in a specified direction, acquiring the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction; Based on the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction, data of the first search image block in the first reference frame and data of the second search image block in the second reference frame are read from the specified storage area.

6. The method according to claim 5, characterized in that The reading, based on the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction, data of the first search image block in the first reference frame and data of the second search image block in the second reference frame from the specified storage area specifically includes: When the center position of the first search image block in the specified direction and the center position of the second search image block in the specified direction are within the first specified range, reading data of the first search image block and the second search image block in the specified storage area; When the center position of the specified direction of the first search image block and the center position of the specified direction of the second search image block are within the second specified range, the area range of the first search image block and the second search image block are determined based on the downsampling processing method; data is read from the specified storage area based on the area range of the first search image block and the area range of the second search image block, and an upsampling operation is performed on the read data to restore the data amount of the first search image block and the data amount of the second search image block, so that the data amount of the first search image block and the data amount of the second search image block are the same as the data amount of the image block to be interpolated.

7. A display device, characterized in that: include: Display, processor and memory; The display is used to display the image; The memory is used to store the processor executable instructions; The processor is configured to execute the instructions to implement the frame insertion processing method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: include: When the instructions in the computer-readable storage medium are executed by a display device, the display device is enabled to execute the frame insertion processing method according to any one of claims 1 to 6.

9. A computer program product, characterized in that include: Computer programs; When the computer program is executed by a processor, the frame insertion processing method according to any one of claims 1 to 6 is implemented.

Images