Image processing system, control device, control method, and image processing method

Abstract

Embodiments of the present application provide an image processing system and a control device, a control method and an image processing method thereof. The image processing system comprises at least two image processing modules, which process image frames and write processed data into a memory. The control device comprises a consistency synchronization module configured to set state information of a data unit of the image frame for the image processing module, and store the state information of the data unit of the image frame for the image processing module, wherein the state information of the data unit for the image processing module is set to a first value in a case where data processed by the image processing module on the data unit is written into the memory.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image processing systems and their control devices, control methods, and image processing methods. Background Technology

[0002] With the continuous advancement of chip manufacturing processes and related technologies, the number of transistors that can be accommodated within a unit area of ​​a chip is constantly increasing. The computational density and data bandwidth of integrated circuit chips are also becoming increasingly higher, supporting ever-increasing video image resolution, frame rate, bit depth, and the number of parallel video streams. Therefore, the following corresponding technologies exist in the current field of video processing (including computation and transmission):

[0003] 1. In response to the ever-increasing volume of data in the field of video computing, and due to the need for compatibility with multiple standards, interfaces, and bandwidth limitations, various video data compression methods have emerged during the video data computing and transmission process. These include ARM Frame Buffer Compression (AFBC) in the video processing field, and Display Stream Compression (DSC) technology used in the Display Port (DP) video interface standard released by the Video Electronics Standards Association (VESA) in the video transmission field. Considering compatibility, many solutions still use the original, uncompressed video data for processing.

[0004] 2. Video computing and transmission involve multiple processing steps. The various video computing modules use frame-level synchronization, meaning subsequent calculations can only proceed after the entire video data frame has been calculated and written to the frame. Chips with video processing capabilities often use a central processing unit (CPU) to uniformly control and synchronize the relevant video computing modules, performing parallel decoding, analysis, calculation, display transmission, or encoding of multiple video data streams. This requires the CPU to respond to interrupts promptly and perform timely task control and scheduling.

[0005] 3. Monitors evolved from the raster scanning sequence of the original cathode ray tubes. Therefore, the transmission and display of video data often require processing the video data according to the raster scanning sequence.

[0006] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0007] The inventors of this application have discovered that, in the aforementioned prior art, there are at least the following technical problems: the CPU performs unified control and synchronization of various modules in video processing. Currently, the various modules in video processing use frame-level synchronization. As the synchronization method becomes more refined and the chip size continues to increase, the workload involved in CPU control and synchronization increases dramatically, and the CPU's processing power becomes insufficient, thereby affecting the performance of the video chip.

[0008] To address at least the aforementioned technical problems or similar technical issues, embodiments of this application provide an image processing system and its control device, control method, and image processing method. In this control device, the status information of the data units of an image frame for each image processing module can be set and recorded. This status information enables synchronization between the image processing modules, reducing the workload associated with controlling the synchronization of the image processing modules and ensuring the performance of the image processing system.

[0009] This application provides a control device for an image processing system. The image processing system includes at least two image processing modules, which process image frames and write the processed data into a memory. The control device includes a consistency synchronization module, which is configured to:

[0010] The data unit of the image frame is set to the status information of the image processing module; and

[0011] The data unit storing the image frame relates to the status information of the image processing module.

[0012] Wherein, when the data processed by the image processing module is written into the memory, the status information of the data unit for the image processing module is set to a first value.

[0013] This application also provides an image processing system, which includes at least two image processing modules. These modules process image frames and write the processed data into a memory. The image processing system also includes the control device described in the above embodiments.

[0014] The data processed by the at least two image processing modules have the same data format.

[0015] In at least one embodiment, at least two image processing modules process the image frame according to a predetermined sequence, wherein, if the state information of the image processing module that processes the image frame first is a first value, the image processing module that processes the image frame second reads the data corresponding to the data unit that has been processed first from the memory and performs the second processing.

[0016] This application also provides a display system, including an image processing system and a display control system. The image processing system includes at least two image processing modules and the control device described in the above embodiments.

[0017] The display control system receives data output by the display output module of the control device.

[0018] When the data received by the display control system corresponds to a complete image frame, a display control signal is output, which controls the display to show the complete image frame. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0020] Figure 1 This is a schematic diagram of the system architecture for video compression, memory storage, video computing, and video transmission and display in the prior art;

[0021] Figure 2 This is a schematic diagram of the video computing process in the prior art;

[0022] Figure 3 This is a schematic diagram illustrating the time delay in video display in existing technology;

[0023] Figure 4 This is a schematic diagram of an image processing system according to an embodiment of the first aspect;

[0024] Figure 5 This is a schematic diagram illustrating the status information of each data block of the image frame set in operation 401 for each image processing module.

[0025] Figure 6 This is a schematic diagram of a status record table;

[0026] Figure 7 This is a schematic diagram of data processing in image processing system 4;

[0027] Figure 8 This is a schematic diagram of data transmission in image processing system 4;

[0028] Figure 9 This is a schematic diagram of a display control system according to an embodiment of the second aspect;

[0029] Figure 10 This is a schematic diagram of a display system according to an embodiment of the second aspect;

[0030] Figure 11 This is an illustration of a control method for an image processing system according to a third aspect embodiment;

[0031] Figure 12 This is a schematic diagram of an image processing method according to an embodiment of the third aspect;

[0032] Figure 13 This is a schematic diagram of a display control method according to an embodiment of the third aspect. Detailed Implementation

[0033] Referring to the accompanying drawings, the foregoing and other features of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of this application can be adopted. It should be understood that this application is not limited to the described embodiments; rather, it includes all modifications, variations, and equivalents falling within the scope of the appended claims. Various embodiments of this application are described below with reference to the accompanying drawings. These embodiments are merely exemplary and not intended to limit the scope of this application.

[0034] In the embodiments of this application, the terms "first," "second," "upper," "lower," etc., are used to distinguish different elements by their names, but do not indicate the spatial arrangement or temporal order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in connection with the application and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0035] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0036] First aspect of the embodiments

[0037] Before describing the embodiments of the first aspect, the working principles of existing image processing (e.g., video processing) and display will be briefly explained below.

[0038] Monitors evolved from the raster scanning sequence of the original cathode ray tubes. In the fields of video transmission and display, interfaces such as Composite Video Broadcast Signal (CVBS), Component Video Rendering (YPbPr), High Definition Multimedia Interface (HDMI), and DisplayPort (DP) often require video data to be processed in raster scanning sequence.

[0039] As chip computing power increases, the data computation order of raster order becomes a constraint. For example, basic video encoding and decoding calculations use block (e.g., data block) arrangements such as 4x4, 8x8, 16x16, 32x32, 64x64, etc. Graphics processors (GPUs) adopt many-core and parallel computing structures, where each computing core can perform calculations relatively independently in a block-based manner. Traditional frame synchronization and scanning order for multi-level data synchronization between different computing modules will cause computational latency and display delay.

[0040] The following explanation will be based on typical examples.

[0041] Figure 1 This is a schematic diagram of a typical video processing chip. For example... Figure 1 As shown, in a typical example, the chip related to video processing (e.g., a system-on-a-chip) may include: a video decoding module A, a GPU module B, a video processing module C, and a video display output module D. The video decoding module A, GPU module B, video processing module C, and video display output module D are connected to memory 100 via a system bus. Memory 100 may be RAM, for example, double-data-rate synchronous dynamic random access memory (DDR).

[0042] The video processing procedures and data in modules A, B, and C are sequentially dependent. Specifically, video decoding module A uses the intra-frame compression format compress_a (com_a), GPU module B uses the intra-frame compression format compress_b (com_b), video processing module C uses the compression format compress_c (com_c), and the video display output module D uses the compression format compress_d (com_d) for its video interface.

[0043] Modules A, B, and C use `compress_a / b / c`, each requiring independent memory space to store video data (i.e., `compress_a_data`, `compress_b_data`, and `compress_c_data`), and need to exchange data for compression format conversion. For example, if module A uses the data format `compress_a`, it stores the data in memory (DDR) through the compression submodule `com_a`. Furthermore, its own independent decompression submodules `dec_b` and `dec_c` need to read the compressed data from modules B and C in memory and decompress it. Similarly, when modules B and C need to use data from module A, they need to read the data in `compress_a_data` (format `compress_a`) from memory and use their respective independent decompression submodule `dec_a` to decompress the `compress_a` data.

[0044] Module D may use the video data calculated by modules A / B / C for display output. Therefore, it is necessary to use the decompression submodules dec_a / b / c to decompress the data and then recompress it via com_d before transmitting it to the monitor.

[0045] Theoretically, if a video processing chip has N modules and uses n different data formats, where each module has m... n Given the dependencies between the compressed data, n different spaces need to be allocated in memory to store data of different formats, and Num decompression submodules and n compression submodules are used. The value of Num can be obtained by the following equation (1).

[0046]

[0047] Figure 1 It uses 4 different modules with 4 different data formats, occupies 3 types of memory, and uses a total of 12 decompression submodules and 4 compression submodules.

[0048] From a storage perspective, processing modules A / B / C use different data formats, so each needs to independently store the required multiple frames of data. The storage space must be divided into several independent areas, and if there are data dependencies between them, data conversion is required, consuming additional storage space for conversion.

[0049] like Figure 2As shown, taking video decoding (e.g., processing by module A) as an example, a single video decoding process typically calls 16 frames of video reference data, denoted as An, and saves the decoded 2 frames of video data, denoted as A2Bn. Taking simple GPU video computation (e.g., processing by module B) as an example, its geometric calculations, rendering calculations, rasterization, and other processes become increasingly complex and consume a large amount of cached video frame-level data, denoted as Bn. Let's tentatively set Bn to 5 frames, and save the calculated result B2Cn as 2 frames. The data involved in the video processing module (e.g., module C) is denoted as Cn, for example, 2 frames. In this example, completing a full video decoding process, from GPU rendering to video processing computation, requires storing at least 16 + 2 + 5 + 2 + 3 = 27 frames of data.

[0050] As can be seen from the above, due to the use of multiple different data formats, the different compression methods used in the user video data storage process during chip design and implementation inevitably occupy more storage space. The need for multiple conversions of different compressed data significantly reduces processing efficiency and increases computational latency. Furthermore, since the CPU is used to synchronize data between modules and control the module computation order, large computational loads can lead to a decrease in CPU performance, directly affecting the overall computational efficiency of the chip.

[0051] Furthermore, video processing often employs frame synchronization, requiring subsequent calculations to be performed only after the entire video data frame has been calculated and written to the frame. For example... Figure 3 As shown, the display (i.e., the display receiver) often requires pre-storing one frame of data to ensure display timing. For example, when displaying the nth frame of data, the data of the (n+1)th frame of data is pre-transmitted and stored; in addition, when transmitting the (n+1)th frame of data, the data of the (n+2)th frame of data can be processed.

[0052] From a system latency perspective, multiple independent processing stages combined with multi-level data format conversions result in longer latency. Theoretically, if each processing module in an N-level processing module generates a corresponding latency of Nn frames (e.g., module A generates a latency of A`n frames, module B generates a latency of B`n frames, module C generates a latency of C`n frames, module D generates a latency of D`n frames, and so on), and there are dependencies between the processing of each module (e.g., the processing of a later module depends on the processing result of a previous module), then the latency of the entire processing is A`n + B`n + C`n + D`n + ... frames. For example, a minimum latency of 4 frames is required in a four-level video processing process, plus an additional 2 frames of latency from intermediate conversion processing; therefore, a total latency of at least 6 frames will be generated.

[0053] An embodiment of the first aspect of this application provides an image processing system and a control device for the image processing system.

[0054] Figure 4 This is a schematic diagram of an image processing system according to an embodiment of the first aspect. For example... Figure 4 As shown, the image processing system 4 includes an image processing module 41 and a control device 42.

[0055] In at least one embodiment, the image processing module 41 can process image frames, such as those from video, i.e., the image frame can be a video frame. The image processing module 41 can write the processed data into the memory 40. For example, the image processing module 41 writes the processed data into the memory 40 via a system bus. The memory 40 can be RAM, and its type can be DDR.

[0056] The number of image processing modules 41 can be at least two. Figure 4 In this embodiment, there are three image processing modules 41, which are referred to as image processing module A, image processing module B, and image processing module C in the description that follows. It should be noted that the embodiments of this application are not limited to this, and the number of image processing modules 41 may be four or five or more.

[0057] In at least one embodiment, image processing module A may be, for example, a video decoding module, image processing module B may be, for example, a GPU module, and image processing module C may be, for example, a video processing module. These are merely examples; image processing modules A, B, and C may also perform other types of processing on image frames.

[0058] like Figure 4 As shown, in at least one embodiment, the control device 42 may include a consistency synchronization module 421. The consistency synchronization module 421 may be configured to perform the following operations:

[0059] Operation 401: Set the status information of the image frame's data unit for the image processing module; and

[0060] Operation 402: The data unit storing the image frame is for the status information of the image processing module.

[0061] In operation 401, when the data processed by the image processing module 41 is written to the memory 40, the status information of the data unit for the image processing module 41 can be set to a first value.

[0062] For example, for any data unit in an image frame, this data unit has state information a for image processing module A, state information b for image processing module B, and state information c for image processing module C; in operation 401, if image processing module A processes the data unit and successfully writes the processed data into memory 40, then state information a is set to a first value; if image processing module B processes the data unit, but the processed data has not yet been written into memory 40 or the writing fails, then state information b is not set to the first value (for example, state information b maintains its original value, which is, for example, a value different from the first value); if image processing module C does not process the data unit, then state information c is not set to the first value (for example, state information c maintains its original value, which is, for example, a value different from the first value). That is, the consistency synchronization module 421 needs to set and record the state information of the data unit for different image processing modules.

[0063] According to an embodiment of the first aspect of this application, the consistency synchronization module 421 can set and record the status information of the data units of the image frame for each image processing module. Thus, it can record the processing of each image unit by each image processing module and the storage of the processed data, thereby supporting synchronization between the image processing modules. For example, each image processing module can determine the processing and data storage status of each image unit based on the status information, and thus determine the processing action of the image processing module for that image unit. Therefore, there is no need to perform complex synchronization control on multiple image processing modules, which can reduce the workload of the image processing system and ensure the performance of the image processing system.

[0064] In at least one embodiment, a data block in an image frame can be defined as a data unit; that is, an image frame can contain multiple data blocks. The size of the data block can be flexibly configured. For example, a data block can be, but is not limited to, 4x4 pixel units, 8x8 pixel units, 16x16 pixel units, 64x64 pixel units, etc., and a pixel unit may contain more than one pixel. The configuration of the data block size is based on factors such as predetermined delay precision and / or the processing capabilities of each image processing module.

[0065] When a data block in an image frame is defined as a data unit, the status information of the data unit can include the status information of each data block. Figure 5 This is a schematic diagram illustrating the status information of each data block of the image frame set in operation 401 for each image processing module. For example... Figure 5 As shown, the status information for each data block of an image frame relative to each image processing module can include:

[0066] Operation 501: Receive location number;

[0067] Operation 502: Receive a write confirmation signal from memory 40 for the data block corresponding to the location number; and

[0068] Operation 503: Set the data block corresponding to the position number to the first value for the status information of the image processing module.

[0069] In operation 501, the position number received by the consistency synchronization module 421 can come from the image processing module 41. The position number can be used to indicate which data block in the image frame the image processing module 41 processed. The position number can include information about the image processing module 41, information about the image frame (e.g., frame number, etc.), and the position information of the data block in the image frame. For example, after processing the data block at position (i, j) in image frame x, image processing module A sends the position number to the consistency synchronization module 421. This position number includes: information about image processing module A (e.g., identification information of image processing module A), identification information of image frame x (e.g., frame number), and the position information (i, j) of the data block in the image frame. Furthermore, when writing the processed data into the memory 40, the image processing module 41 can also write information related to the position of the corresponding data unit (e.g., the frame number of the image frame and the position information (i, j) of the data block in the image frame) into the memory 40.

[0070] In operation 502, the write confirmation signal received by the consistency synchronization module 421 can come from the memory 40. For example, when the image processing module 41 successfully writes the data processed for the data block into the memory 40, the memory 40 sends a write confirmation signal to the consistency synchronization module 421.

[0071] In operation 503, upon receiving a write confirmation signal, the consistency synchronization module 421 sets the status information of the data block according to the position number received in operation 501. For example, in operation 503, based on the position number received in operation 501, the consistency synchronization module 421 sets the status information of the data block with position information (i, j) in image frame x to a first value for the image processing module A. In operation 503, the first value is, for example, 1.

[0072] In at least one embodiment, an image frame can be defined as a data unit. Therefore, the status information of the data unit can include the status information of the image frame. When setting the status information of an image frame, the consistency synchronization module 421 can set the status information of the image frame for the image processing module 41 to a first value if the image processing module 41 has completed processing of all pixel units in the image frame and successfully written the processed data into the memory 40.

[0073] In at least one embodiment, the data unit may include both a data block in the image frame and the image frame itself. Thus, the consistency synchronization module 421 can, according to... Figure 5 Based on the method of setting the status information of the data block, the status information of the image frame for each image processing module 41 is further set according to the status information of each data block in the image frame for each image processing module 41.

[0074] For example, if the status information of each data block in the image frame is set to the first value for the image processing module A, then the status information of the image frame for the image processing module A is set to the first value. That is, if the image processing module A in the image processing system 4 processes each data block in the image frame and successfully stores the processed data into the memory 40, then the status information of the image frame for the image processing module A is set to the first value.

[0075] For example, when the status information of each data block in the image frame for the image processing module B is set to the first value, the status information of the image frame for the image processing module B is set to the first value. That is, when the image processing module B in the image processing system 4 processes each data block in the image frame and successfully stores the processed data into the memory 40, the status information of the image frame for the image processing module B is set to the first value.

[0076] For example, when the status information of each data block in the image frame for the image processing module C is set to the first value, the status information of the image frame for the image processing module C is set to the first value. That is, when the image processing module C in the image processing system 4 processes each data block in the image frame and successfully stores the processed data into the memory 40, the status information of the image frame for the image processing module C is set to the first value.

[0077] Furthermore, when the status information of each data block in the image frame for each image processing module 41 is set to the first value, it can be considered that the entire frame data of the image frame is fully ready. That is, when all image processing modules A, B, and C in the image processing system 4 have processed each data block in the image frame and successfully stored the processed data in the memory 40, the consistency synchronization module 421 can determine that the entire frame data of the image frame is fully ready. For example, image processing module B can only process after image processing module A has finished processing, and image processing module C can only process after image processing module B has finished processing. When the status information of the image frame for image processing module C is set to the first value, it means that all image processing modules A, B, and C in the image processing system 4 have processed each data block in the image frame and successfully stored the processed data in the memory 40. At this time, the consistency synchronization module 421 can determine that the entire frame data of the image frame is fully ready.

[0078] If a portion of the status information of each data block in the image frame for each image processing module is not set to the first value, that is, all image processing modules 41 of the image processing system 4 have not fully completed the processing of all data blocks in the image frame and stored the processed data into the memory 40, the consistency synchronization module 421 can determine that part of the data of the image frame is ready.

[0079] If none of the data blocks in the image frame have their status information set to the first value for each image processing module, that is, if no image processing module 41 in the image processing system 4 completes the processing of any data block in the image frame and stores the processed data into the memory 40, the consistency synchronization module 421 can determine that the data of the image frame is not ready at all.

[0080] After the entire frame data of the image frame is synchronized (e.g., after the entire frame data of the image frame is sent), the status information of the image frame for each image processing module can be cleared (e.g., the status information of the image frame for each image processing module can be set to a value different from the first value). In addition, the status information of each data block in the image frame can also be cleared (e.g., the status information of each data block in the image frame for each image processing module can be set to a value different from the first value).

[0081] In this application, when a data unit includes both a data block in an image frame and the image frame itself, the state information of the data unit includes the state information of the image frame and the state information of the data block in the image frame. Therefore, the state information of the data unit can be multi-level; that is, the state information of the image frame is the state information of the previous level, and the state information of the data block in the image frame is the state information of the next level. Furthermore, each data block can be further divided into multiple sub-data blocks, and each sub-data block can also be further divided. Thus, the image frame can be divided into more levels, and each level can have its own state information, thereby expanding the number of levels of state information. The state information of the previous level can be set based on the state information of the next level, or it can be directly set based on the processing and storage conditions of each image processing module 41 for that previous level.

[0082] In at least one embodiment, the state information can be stored in the storage unit of the consistency synchronization module 421 ( Figure 4 (Not shown). The size and type of this storage unit can be set according to the scale of data to be stored (e.g., the scale of status information data). For example, when the amount of status information is large (e.g., in a scenario of synchronization between multiple video modules), the storage unit can be set as internal static memory, thereby increasing storage space and reducing the chip area occupied. Alternatively, when the amount of status information involved is small, the storage unit can be set as a register.

[0083] In at least one embodiment, the state information of the data unit can be stored in the form of a state record table. Figure 6 This is a schematic diagram of a status log table. For example... Figure 6 As shown, in a specific example, the storage unit of the consistency synchronization module 421 stores a first state record table 61 and a second state record table 62. The first state record table 61 records the state information of N image frames; the second state record table 62 records the state information of each data block in each of the N image frames. N is, for example, 32. The storage unit of the consistency synchronization module 421 can have registers and static memory, wherein the first state record table 61 can be stored in a register, and the second state record table 62 can be stored in static memory.

[0084] like Figure 6As shown in the second state record table 62: A white box indicates that the data block has not been processed by any of the image processing modules A, B, or C. For example, the state information corresponding to a white box can be represented by 0. A light gray box indicates that the data block has been processed by image processing module A, and the processed data has been successfully written to memory 40; that is, the state information of this data block for image processing module A is the first value. A dark gray box indicates that the data block has been processed by image processing module B, and the processed data has been successfully written to memory 40; that is, the state information of this data block for image processing module B is the first value. A black box indicates that the data block has been processed by image processing module C, and the processed data has been successfully written to memory 40; that is, the state information of this data block for image processing module C is the first value. The first value is, for example, 1.

[0085] like Figure 6 As shown, in the first state record table 61: A white box indicates that none of the data blocks of the image frame have been processed by any of the image processing modules A, B, or C. For example, a white box means that the state information of the image frame for each image processing module can be 0, indicating that the data of the image frame is not ready at all. A light gray box indicates that all the data blocks of the image frame have been processed by image processing module A, and the processed data has been successfully written to memory 40. That is, the state information of the image frame for image processing module A is the first value, and the data of the image frame is partially ready. A dark gray box indicates that all the data blocks of the image frame have been processed by image processing module B, and the processed data has been successfully written to memory 40. That is, the state information of the image frame for image processing module B is the first value, and the data of the image frame is partially ready. A black box indicates that all the data blocks in the image frame have been processed by image processing module C, and the processed data has been successfully written to memory 40. That is, the state information of the image frame for image processing module C is the first value, and the data of the image frame is fully ready. The first value is, for example, 1.

[0086] like Figure 6 As shown, each data block is 64x64 pixels in size. Taking a 1920x1080 resolution image as an example, for the processing of image processing modules A, B, and C, a total of 32x3x(1920 / 64)x(1088 / 64) = 32x3x510 bits of status information needs to be recorded. Here, 3 represents the number of status information records needed for each of processing modules A, B, and C, and 32 represents 32 image frames. This 32x3x510 bits of status information (i.e., the information in the second status record table 62) can be stored in static memory. The status information in the first status record table 61 is 32x3 bits and can be stored in a register.

[0087] In at least one embodiment, the consistency synchronization module 421 is also configured to perform the following operations:

[0088] Operation 403: When the status information of the data unit changes, send a status update signal to at least one image processing module 41; and / or

[0089] Operation 404: Receive a status query signal from at least one image processing module 41 and provide feedback on the status information of the data unit indicated by the status query signal.

[0090] Active synchronization can be achieved through operation 403. Specifically, a status update signal can be sent to the post-processing graphics processing module 41. For example, when the status information of a certain data block for image processing module B is set to a first value (i.e., when it changes from a value different from the first value to the first value), the consistency synchronization module 421 can send a status update signal to image processing module C. Thus, image processing module C can read the data corresponding to that data block from memory 40 and perform processing.

[0091] Furthermore, this application is not limited to this; the consistency synchronization module 421 can also send status update signals to other image processing modules, for example, sending status update signals to each image processing module. The consistency synchronization module 421 can also send status update signals to the display output module 422 (described later) (e.g., Figure 4 As shown, the display output module 422 can transmit the data block after determining that the data block has completed all processing and has been written into the memory 40.

[0092] In at least one embodiment of this application, a change in the state information of a data unit can refer to a change in the state information from a value different from the first value to the first value, or a change in the state information from the first value to a value different from the first value. For example, when the state information of a data unit is cleared, the state information changes from the first value to a value different from the first value. At this time, the consistency synchronization module 421 can send a state update signal to the graphics processing module 41 (e.g., the image processing module 41 in preprocessing, or the image processing module 41 in postprocessing, or all image processing modules 41).

[0093] When the data corresponding to a data unit is not used by the graphics processing module 41 in the post-processing stage, the status information of the data unit can be cleared. In addition, the consistency synchronization module 421 can be configured to implement different clearing methods. For example, the status information at the frame level can be cleared, that is, when the data corresponding to an image frame is not used by the image processing module 41 in the post-processing stage, the status information of the image frame for the pre-processing module is cleared. For another example, the status information of two or more data blocks can be cleared together (i.e., multiple blocks are cleared together).

[0094] Passive synchronization can be achieved through operation 404. The status query signal can come from the image processing module 41, and the status query signal can include the location information of the data unit (e.g., location number, etc.). The consistency synchronization module 421 can provide feedback on the status information of the data unit indicated by the status query signal. For example, the image processing module B sends a status query signal to the consistency synchronization module 421, and the status query signal includes the location information of the data block (e.g., the location information can include: image frame and the position of the data block in the image frame, etc.). The synchronization module 421 can provide feedback on the status information of the data block for each image processing module to the image processing module B.

[0095] In operation 404, when the status information of a data unit includes the status information of an image frame and the status information of a data block in the image frame: if part of the data of an image frame is ready, then the consistency synchronization module 421 can provide feedback on the status information of the data unit (e.g., data block) indicated by the status query signal; if the entire frame of data of the image frame is fully ready or the data of the image frame is not ready at all, then the consistency synchronization module 421 can provide feedback on the status information of the image frame to which the data unit indicated by the status query signal belongs (e.g., if the entire frame of data of the image frame is fully ready, the status of the image frame relative to the image processing module C can be provided as a first value; if the data of the image frame is not ready at all, the status of the image frame relative to the image processing module A can be provided as a value different from the first value), that is, the consistency synchronization module 421 may not query and provide feedback on the status information at the data block level.

[0096] like Figure 4 As shown, in at least one embodiment, the control device 42 may further include a display output module 422. The display output module 422 may be configured to perform the following operations:

[0097] Operation 411: Based on the status information stored in the consistency synchronization module 421, determine the data units that meet the predetermined conditions as the data units to be output; and

[0098] Operation 412: Read the data corresponding to the position information of the data unit to be output from the memory 40 and output it.

[0099] In operation 411, the predetermined conditions include: the status information of the data unit for all image processing modules is the first value.

[0100] For example, if a data block has a first value for the status information of image processing module A, a first value for the status information of image processing module B, and a first value for the status information of image processing module C, then the data block is a data block to be output (i.e., a data unit to be output).

[0101] In operation 412, according to the position information of the data unit to be output, the corresponding data is read from the memory 40 and output. For example, the display module 422 can output the data to the display terminal, and the display terminal can display the image corresponding to the data.

[0102] By operating 411 and 412, the display output module 422 can send data in data units, for example, in data blocks (i.e., a portion of an image frame), thereby enabling out-of-order data transmission instead of sending data in the order of the displayed rows as in the prior art.

[0103] In operation 411, the display output module 422 can set the order of reading and outputting the data units to be output according to the order in which the data units meet the predetermined conditions.

[0104] For example, at time T1, the status information of image processing modules A, B, and C for data block 1 all reaches the first value; at time T2, the status information of image processing modules A, B, and C for data block 2 also reaches the first value. Since T1 is earlier than T2, the display output module 422 reads and outputs the data corresponding to data block 1 first. Therefore, data can be transmitted according to the order in which it was stored in memory 40, thereby improving data transmission speed, reducing latency, and saving data storage space.

[0105] In the image processing system 4 of the first aspect embodiment, the at least two image processing modules 41 can process image frames according to a predetermined order. For example, the processing of image processing module A is before the processing of image processing module B, and the processing of image processing module C is after the processing of image processing module B.

[0106] When the status information of the previously processed image processing module 41 is a first value for a data unit in an image frame, the subsequent image processing module 41 reads the previously processed data corresponding to that data unit from the memory and performs subsequent processing. Thus, the image processing system 40 can perform out-of-order processing of image frames; that is, after processing a data block (i.e., data unit), the next processing module can immediately perform subsequent processing without frame-level waiting, eliminating the need for complex synchronization control, greatly reducing reliance on the CPU, and further accelerating the image processing flow.

[0107] For example, for a certain data block: image processing module A processes the data of the data block and stores the processed data in memory 40; consistency synchronization module 421 sets the status information of the data block for image processing module A to a first value; subsequently, image processing module B reads the data of the data block obtained after processing by image processing module A from memory 40, processes the data, and stores the processed data in memory 40; consistency synchronization module 421 sets the status information of the data block for image processing module B to a first value; subsequently, image processing module C reads the data of the data block obtained after processing by image processing module B from memory 40, processes the data, and stores the processed data in memory 40.

[0108] In at least one embodiment of the image processing system 4, the data processed by the at least two image processing modules 41 has the same data format. For example, the data processed by image processing modules A, B, and C have the same data format. Compared with the prior art where different image processing modules use different data formats, this application reduces storage space and data transmission bandwidth by adopting a unified data format, eliminating the need for data conversion between different formats.

[0109] Furthermore, this application is not limited to this. The data processed by the at least two image processing modules 41 may not have the same data format. In this case, the image processing system 4 needs to perform processing involving the conversion between different data formats and corresponding storage space.

[0110] The following is a specific example illustrating the data processing and transmission method of the image processing system 4 of this application.

[0111] Figure 7 This is a schematic diagram of data processing in image processing system 4. Figure 8 This is a schematic diagram of data transmission in image processing system 4. Figure 7 and Figure 8In one example, at least two image processing modules 41 can process an image frame in a predetermined order. For example, the data processed by image processing module A is used for processing by image processing module B, and the data processed by image processing module B is then processed by image processing module C. When all data blocks in an image frame have been processed by image processing module C and written into memory 40, the status information of the image frame is set to a first value.

[0112] exist Figure 7 and Figure 8 In the respective second state record tables shown, white boxes indicate that the data block has not been processed by any of the image processing modules A, B, or C. For example, the state information corresponding to a white box can be represented by 0. Light gray boxes indicate that the data block has been processed by image processing module A, and the processed data has been successfully written to memory 40; that is, the state information of the data block for image processing module A is the first value. Dark gray boxes indicate that the data block has been processed by image processing module B, and the processed data has been successfully written to memory 40; that is, the state information of the data block for image processing module B is the first value. Black boxes indicate that the data block has been processed by image processing module C, and the processed data has been successfully written to memory 40; that is, the state information of the data block for image processing module C is the first value. The first value is, for example, 1.

[0113] like Figure 7 and Figure 8 As shown, a workflow for data processing and transmission in image processing system 4 includes:

[0114] Operation 71: In time segment Time0, image processing module A processes the data block with position information (0, 1), (1, 0), (h, j), (1, n-1) in image frame Frame 0, and writes the processed data to memory 40. After receiving the write confirmation signal from memory 40, consistency synchronization module 421 sets the status information of the data block with position information (0, 1), (1, 0), (h, j), (1, n-1) to the first value. For example... Figure 7 As shown in the light gray box.

[0115] In operation 72, under active synchronization, the consistency synchronization module 421 sends a state update signal to the image processing module B to indicate that the data blocks (0, 1), (1, 0), (h, j), (1, n-1) of Frame 0 have changed in relation to the state information of the image processing module A. The image processing module B can choose to observe the first state record table and / or the second state record table. This example uses a data block-level synchronization mechanism. In time segment T1, the image processing module B reads the data corresponding to the four data blocks (0, 1), (1, 0), (h, j), (1, n-1) from the memory 40, processes them, writes the processed data back to the memory 40, and sets the state of (0, 1), (1, 0), (h, j), (1, n-1) of Frame 0 to the first value for the state of the image processing module B, such as... Figure 7 As shown in the dark gray block. Within the same time segment Time 1, image processing module A also completed the processing and writing of the (o, p) and (0, m-1) data blocks of Frame0, as shown in the dark gray block. Figure 7 As shown in the dark gray box.

[0116] In operation 73, under active synchronization, image processing module C receives a status update signal for image processing module B sent by consistency synchronization module 421, and then performs data reading, processing, and writing operations to memory 40. For example, during time segment TimeM, image processing module C completes processing on certain data blocks and writes the processed data to memory 40. Consistency synchronization module 421 sets the status information of these data blocks to a first value, such as... Figure 7 As shown in the black box.

[0117] Operation 74, such as Figure 8 As shown, the transmission of data processed and written by the image processing module C can employ out-of-order transmission of data blocks based on the order in which processing is completed. For example, in Figure 8 In the image processing module C, the data blocks numbered 34, 27, 06, and 12 of the image frame with frame number Frame n are processed sequentially and the processed data is written to the memory 40. Therefore, the display output module 422 transmits the data corresponding to the data blocks numbered 34, 27, 06, and 12 sequentially.

[0118] On the display end, when the display system receives the data corresponding to the data blocks numbered 34, 27, 06, and 12 in sequence, it also receives the position information of the data blocks (this position information may include: the frame number Frame n of the image frame and the number information of the data block in the image frame, etc.). Based on this information, after receiving the data of a complete image frame, the display can be controlled to process the image of that image frame.

[0119] In the above example, the minimum delay between image frame processing and display can be controlled to be the sum of the processing delay of 3 data blocks (e.g., caused by image processing modules A, B, and C respectively) and the display transmission delay of 1 data block.

[0120] In summary, the embodiments of the first aspect of this application have the following beneficial effects:

[0121] 1. Taking video decoding, GPU rendering computation and image processing computation as examples, when dividing the process in this application, since the video data format in memory has been unified, we can consider designing more from the perspective of video flow. We can reduce the space of several frames of data that need to be converted between A / B / C modules, and reduce or be free from the constraints of A / B / C data format.

[0122] 2. Compared to existing methods of storing video data in memory using different compression techniques, this application reduces storage space and data transmission bandwidth, and decreases chip area, by adopting a unified intra-frame compression format for video data and storing it in memory using a unified compressed or uncompressed method. It also supports uncompressed raw data formats.

[0123] 3. Compared with existing video computing modules that often use frame-level synchronization, which requires the entire video data frame to be calculated and written before subsequent calculation operations can be performed, the modules in this application can actively or passively learn about the update status of relevant video frames or intra-frame data blocks. They can choose to use frame-level or data block-level calculations. The calculation unit that completes one data block can immediately calculate the next data block without waiting for frame-level calculations, greatly reducing the dependence on the CPU and further accelerating the video processing flow.

[0124] 4. Compared with the existing traditional frame synchronization and scanning order, which cause calculation delay and display delay due to multi-level data synchronization between different calculation modules, this application adopts out-of-order transmission of data blocks based on the order of calculation completion. Calculating and transmitting video data according to the order of video processing rather than the line order of display can shorten the delay from calculation to display and save video data storage space.

[0125] Second aspect of the embodiments

[0126] An embodiment of the second aspect of this application provides a display control system and a display system having the display control system.

[0127] Figure 9 This is a schematic diagram of the display control system. For example... Figure 9 As shown, the display control system 9 can receive data output by the display output module and control the display 91 to display image frames.

[0128] When the data received by the display control system 9 corresponds to a complete image frame, it outputs a display control signal, which controls the display 91 to display the complete image frame. This allows for display without adhering to the order of the displayed rows, thereby reducing display delay.

[0129] In at least one embodiment, the display control system 9 can determine whether the received data can be combined into a complete image frame based on the frame number information of the image frame corresponding to the received data and the numbering information of the data unit (e.g., data block) in the image frame.

[0130] Figure 10 This is a schematic diagram of a display system according to an embodiment of the second aspect. For example... Figure 10 As shown, the display system 10 may include an image processing system 101 and a display control system 9.

[0131] The image processing system 101 may have a display output module (not shown), which can send the processed data to the display control system 9. The display control system 9 controls the display based on the data output by the display output module so that the display 91 can display the data.

[0132] In at least one embodiment, the image processing system 101 may be the image processing system 4 described in the first aspect embodiment. That is, all descriptions of the image processing system 4 described in the first aspect embodiment are applicable to the image processing system 101. Thus, at least two image processing modules in the image processing system 101 can perform out-of-order processing on data blocks, and the display output module in the image processing system 101 can output the data corresponding to the data units (e.g., data blocks) to the display control system 9 in the order in which the data units (e.g., data blocks) are processed. When the display control system 9 receives the data of a complete image frame, it controls the display 91 to display the complete image frame.

[0133] Furthermore, this application is not limited to this, and the image processing system 101 may also be different from the image processing system 4 described in the first aspect embodiment.

[0134] Third aspect of the embodiments

[0135] The third aspect of this application provides a control method for an image processing system, corresponding to the control device for the image processing system in the first aspect.

[0136] Figure 11 This is a schematic diagram of the control method of an image processing system, such as... Figure 11 As shown, the control method includes:

[0137] Operation 1101: Set the status information of the data unit of the image frame for each of the image processing modules; and

[0138] Operation 1102: The data unit storing the image frame stores the status information of each of the image processing modules.

[0139] Specifically, when the data processed by the image processing module is written into the memory, the state information of the data unit for the image processing module is set to a first value.

[0140] The third aspect of this application provides an image processing method for an image processing system, corresponding to the image processing system 4 of the first aspect embodiment.

[0141] Figure 12 This is a schematic diagram of the image processing method, such as... Figure 12 As shown, the image processing method includes:

[0142] Operation 1201: At least two image processing modules process the image frame; and

[0143] Operation 1202: Set and store the status information of the data unit of the image frame for each of the image processing modules.

[0144] In operation 1201, at least two image processing modules process the image frame according to a predetermined sequence. When the data unit of the image frame has a first value for the status information of the image processing module that was processed earlier, the image processing module that is processed later reads the data corresponding to the data unit from the memory and performs the later processing.

[0145] The third aspect of this application provides a display control method for a display system, which corresponds to the operation of the display control system 9 of the second aspect embodiment.

[0146] Figure 13 This is a schematic diagram of the display control method, such as... Figure 13 As shown, the display control method includes:

[0147] Operation 1301: When the data received from the display output module of the control device corresponds to a complete image frame, a display control signal is output. This display control signal is used to control the display to show the complete image frame.

[0148] In operation 1301, the display control system 9 can determine that the received data corresponds to a complete image frame based on the frame number information of the image frame corresponding to the received data and the number information of the data unit in the image frame.

[0149] Embodiments of this application also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods in the embodiments of the third aspect.

[0150] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods in the embodiments of the third aspect.

[0151] Embodiments of this application also provide a computer program product comprising a computer program that, when executed by a processor, implements any of the methods in the embodiments of the third aspect.

[0152] The acquisition, storage, use, and processing of data in the various embodiments of this application all comply with the relevant provisions of national laws and regulations.

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A control device for an image processing system, the image processing system comprising at least two image processing modules, the at least two image processing modules processing image frames and writing the processed data into a memory. Its features are, The control device includes a consistency synchronization module and a display output module, wherein the consistency synchronization module is configured as follows: The data units of the image frames are configured to correspond to the status information of each of the image processing modules; and The data units storing the image frames are respectively for the status information of each of the image processing modules. Each of the data units has at least two states, and each of the image processing modules corresponds to at least one state information in each of the data units. in, The status information set and stored by the consistency synchronization module is used to support synchronization between the various image processing modules. When the data processed by the image processing module is written into the memory, the state information of the data unit for the image processing module is set to a first value. The display output module is configured as follows: Based on the status information stored in the consistency synchronization module, the data unit that meets the predetermined conditions is determined as the data unit to be output and output, wherein the predetermined conditions include: all the status information of the data unit is a first value.

2. The control device as described in claim 1, characterized in that, The status information of the data unit includes: The status information of the image frame and / or the status information of the data blocks in the image frame.

3. The control device as described in claim 2, characterized in that, In the case where the data unit includes the data block, Setting the status information of the data unit includes: Receive a location number; the location number includes information about the image processing module, information about the image frame, and the location information of the data block in the image frame; Receive a write confirmation signal from the memory for the data block corresponding to the location number; and The status information of the data block corresponding to the location number for the image processing module is set to the first value.

4. The control device as described in claim 3, characterized in that, When the data unit further includes the image frame, the status information of the data unit includes the status information of the image frame and the status information of the data blocks within the image frame. Setting the status information of the data unit further includes: Based on the status information of each data block in the image frame for each image processing module, the status information of the image frame for each image processing module is set as the status information of the image frame.

5. The control device as described in claim 1, characterized in that, The consistency synchronization module is also configured to: In the event of a change in the state information, a state update signal is sent to at least one of the image processing modules; and / or Receive a status query signal from at least one image processing module and provide feedback the status information of the data unit indicated by the status query signal.

6. The control device as described in claim 1, characterized in that, The display output module is further configured to read data corresponding to the position information of the data unit to be output from the memory and output it.

7. The control device as described in claim 6, characterized in that, The display output module sets the order of reading and outputting the data units to be output according to the order in which the data units meet the predetermined conditions.

8. An image processing system, comprising at least two image processing modules, wherein the at least two image processing modules process image frames and write the processed data into a memory. Its features are, The image processing system further includes the control device according to any one of claims 1 to 7. in, The data processed by the at least two image processing modules have the same data format.

9. The image processing system as described in claim 8, characterized in that, The at least two image processing modules process the image frames according to a predetermined sequence. in, The data unit of the image frame is configured to handle the first value of the state information of the previously processed image processing module. The image processing module in post-processing reads the data corresponding to the data unit from the memory after the prior processing, and performs the post-processing.

10. A display system, comprising an image processing system and a display control system, characterized in that, The image processing system includes at least two image processing modules and the control device as described in claim 6 or 7. The display control system receives data output by the display output module of the control device. Specifically, when the data received by the display control system corresponds to a complete image frame, a display control signal is output. The display control signal controls the display to show the complete image frame.

11. The display system as claimed in claim 10, characterized in that, The display control system determines that the received data corresponds to a complete image frame based on the frame number information of the image frame corresponding to the received data and the number information of the data unit in the image frame.

12. A control method for an image processing system, the image processing system comprising at least two image processing modules, the at least two image processing modules processing image frames and writing the processed data into a memory. Its features are, The control method includes: The consistency synchronization module sets the data units of the image frames to correspond to the status information of each of the image processing modules; and The consistency synchronization module stores the status information of each data unit of the image frame for each of the image processing modules. Each of the data units has at least two states, and each of the image processing modules corresponds to at least one state information in each of the data units. in, The status information set and stored by the consistency synchronization module is used to support synchronization between the various image processing modules. When the data processed by the image processing module is written to the memory, the status information of the data unit for the image processing module is set to a first value. The control method further includes: The display output module determines the data unit that meets the predetermined conditions as the data unit to be output and outputs it according to the status information. The predetermined conditions include: all the status information of the data unit is a first value.

13. An image processing method for an image processing system, the image processing system comprising at least two image processing modules, the at least two image processing modules processing image frames and writing the processed data into a memory. Its features are, The image processing system further includes a control device according to any one of claims 1 to 7, the control device comprising a consistency synchronization module and a display output module; in, The image processing method includes: The at least two image processing modules process the image frame; and The consistency synchronization module sets the status information of each image processing module for each data unit of the image frame; and stores the status information of each image processing module for each data unit of the image frame. Each data unit has at least two status information, and each image processing module corresponds to at least one status information in each data unit. The status information set and stored by the consistency synchronization module is used to support synchronization between the image processing modules. When the data processed by the image processing module on the data unit is written to the memory, the status information of the data unit for the image processing module is set to a first value. The display output module determines and outputs data units that meet predetermined conditions based on the status information stored in the consistency synchronization module. The predetermined conditions include: all the status information of the data unit is a first value.

14. The image processing method as described in claim 13, characterized in that, The at least two image processing modules process the image frames according to a predetermined sequence. in, The data unit of the image frame is configured to handle the first value of the state information of the previously processed image processing module. The image processing module in post-processing reads the data corresponding to the data unit from the memory after the prior processing, and performs the post-processing.

15. A display control method for a display system, the display system comprising an image processing system and a display control system, characterized in that, The image processing system includes at least two image processing modules and the control device as described in claim 6 or 7, wherein the control device includes a consistency synchronization module and a display output module; The display control method includes: The consistency synchronization module sets the status information of each image processing module for each data unit of the image frame; and stores the status information of each image processing module for each data unit of the image frame. Each data unit has at least two status information, and each image processing module corresponds to at least one status information in each data unit. The status information set and stored by the consistency synchronization module is used to support synchronization between the image processing modules. When the data processed by the image processing module on the data unit is written to the memory, the status information of the data unit for the image processing module is set to a first value. The display output module determines and outputs the data units that meet predetermined conditions based on the status information stored in the consistency synchronization module. The predetermined conditions include: all the status information of the data unit is a first value. When the data received by the display control system from the display output module of the control device corresponds to a complete image frame, the display control system outputs a display control signal. The display control signal is used to control the display to show the complete image frame.

16. The display control method as described in claim 15, characterized in that, The display control system determines that the received data corresponds to a complete image frame based on the frame number information of the image frame corresponding to the received data and the number information of the data unit in the image frame.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 12 to 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 12 to 16.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 12 to 16.

Images