A method, apparatus and medium for scaling image optimization based on a matrix system
By using a matrix-based image scaling optimization method, which incorporates sharpening and smoothing processes, the imbalance between image quality and resource consumption in FPGA scaling algorithms is resolved, thereby improving image quality and achieving efficient resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU BAOLUN ELECTRONICS CO LTD
- Filing Date
- 2025-03-07
- Publication Date
- 2026-05-29
AI Technical Summary
Existing FPGA scaling algorithms struggle to balance resource consumption and cost while maintaining image quality, resulting in significant differences in detail between the output image and the original image.
A matrix-based image scaling optimization method is adopted. Through sharpening and smoothing, the image data is processed using a first scaling factor configuration and a second scaling factor configuration respectively. The center point coefficient value is kept dominant, and the edge coefficient ratio is adjusted to adapt to different needs.
It improves image clarity and overall quality, reduces the computational load of subsequent processing steps, achieves optimized balance of image quality, and adapts to different display devices and application scenarios.
Smart Images
Figure CN120107061B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer digital image processing technology, and in particular to a scaling image optimization method, apparatus and medium based on a matrix system. Background Technology
[0002] In applications such as video surveillance, medical imaging, and gaming, optimized processing techniques can make video images clearer and more vivid, significantly improving the user's viewing experience and satisfaction. In particular, the core business functions of video matrix systems are typically implemented using FPGAs, and one of the core tasks of an FPGA is to efficiently execute video image processing algorithms. Therefore, the quality of the final output of scaled data processed by an FPGA, without additional post-processing, directly depends on the accuracy of the processing algorithm used. Currently, mainstream FPGA scaling algorithms include nearest neighbor interpolation, bilinear interpolation, and bicubic scaling. Nearest neighbor interpolation uses the fewest resources but has the worst image quality; bicubic scaling uses the most resources but has the best image quality; bilinear interpolation falls between the two, offering better image quality than nearest neighbor interpolation while using fewer resources than bicubic scaling. Therefore, most mainstream matrix video systems currently use bilinear interpolation.
[0003] However, when using the bilinear scaling algorithm on an FPGA, the output image details are significantly different from the original image. While the bicubic algorithm improves upon this, it is still inferior to the original image. Considering both cost and image quality, optimizing the video quality after scaling on an FPGA requires additional image processing of the backend data. The key to FPGA image processing lies in the scaling algorithm. The higher the algorithm accuracy, the less image data is lost, but the resource consumption and FPGA cost also increase accordingly. Therefore, it is difficult to guarantee the quality of the final output image. Summary of the Invention
[0004] This invention provides a scaling image optimization method, apparatus, and medium based on a matrix system to solve the problem of difficulty in optimizing and balancing video image data to improve image quality.
[0005] To achieve the above objectives, this application provides a scaling image optimization method based on a matrix system, comprising:
[0006] Obtain the raw image data of the video input;
[0007] The original image data is scaled to obtain the first image data;
[0008] Based on the first scaling factor configuration and the second scaling factor configuration, the first image data is sharpened and smoothed sequentially to obtain the target image data; wherein, the first scaling factor configuration is established by maintaining the edge coefficient ratio constant, with the center point coefficient value as the dominant factor; the second scaling factor configuration is established by maintaining the edge coefficient distribution evenly, with the center point coefficient value as the dominant factor.
[0009] In this invention, scaling can adjust the image size as needed to adapt to different display devices or application scenarios. Furthermore, it can reduce the computational load of subsequent processing steps while maintaining image quality, thus improving overall processing efficiency. Based on a first scaling factor configuration and a second scaling factor configuration, the first image data is sharpened and smoothed, respectively. This is a key step in improving image quality. The first scaling factor configuration prioritizes the center point coefficient value, meaning that more attention is paid to information in the central area of the image during processing. This helps preserve the main features and details of the image, resulting in a clearer image. Moreover, by maintaining a constant edge coefficient ratio, the first scaling factor configuration ensures consistency when processing image edges, helping to reduce distortion and blurring at edges and improve overall image quality. The second scaling factor configuration also prioritizes the center point coefficient value, but it is established by maintaining a balanced distribution of edge coefficients. This configuration may help reduce noise and details in the image, making the image smoother and softer. This helps improve the quality of the final output image, meeting different application scenarios and needs.
[0010] Compared to existing technologies, this invention, through scaling, targeted sharpening, and smoothing, can fully leverage the advantages of various technical means, allowing them to complement and coordinate with each other to ensure the quality of the final output image. Therefore, it can solve the problem of difficulty in optimizing and balancing video image data to improve picture quality.
[0011] As a preferred embodiment, the first proportional coefficient configuration is established based on the center point coefficient value, while maintaining a constant edge coefficient ratio, specifically as follows:
[0012] Get the center point coefficient value sent by the client;
[0013] The center point coefficient values are used as the center point coefficients of the sharpening filter matrix, and the difference between the center point coefficient values and the preset maximum values is calculated to obtain the coefficient space;
[0014] In the sharpening filter matrix, the coefficient space is divided according to a preset ratio by controlling the diagonal neighbor points and the four neighbor points to obtain the edge coefficient configuration of the sharpening filter matrix.
[0015] The first scaling factor configuration for sharpening is constituted by the center point coefficient values and the edge coefficient configuration of the sharpening filter matrix.
[0016] In this preferred solution, the client can dynamically adjust the center point coefficient value according to different image content, quality requirements, or application scenarios to adapt to changing processing needs. The coefficient space is divided according to a preset ratio using diagonal and quadruple neighbor points, resulting in more detailed and precise edge coefficient configuration. Furthermore, this detailed edge coefficient configuration helps enhance the image's edge effects, making the image appear clearer and sharper.
[0017] As a preferred embodiment, the second proportional coefficient configuration is established based on the center point coefficient value and by maintaining a balanced distribution of edge coefficients, specifically as follows:
[0018] Get the center point coefficient value sent by the client;
[0019] The center point coefficient values are used as the center point coefficients of the smoothing filter matrix, and the difference between the center point coefficient values and the preset maximum coefficient values is calculated to obtain the coefficient space.
[0020] In the smoothing filter matrix, the coefficient space is divided equally by controlling all neighborhood edge points according to a preset ratio to obtain the edge coefficient configuration of the smoothing filter matrix;
[0021] The second scaling factor configuration for smoothing is constituted by the center point coefficient values and the edge coefficient configuration of the smoothing filter matrix.
[0022] This preferred scheme controls all neighboring edge points to divide the coefficient space equally according to a preset ratio, making the configuration of edge coefficients more balanced and stable; moreover, the balanced configuration of edge coefficients helps to reduce distortion in the image processing process, making the image look more natural and smooth.
[0023] As a preferred embodiment, based on the first scaling factor configuration and the second scaling factor configuration, the first image data is sequentially sharpened and smoothed to obtain the target image data, specifically as follows:
[0024] Pixel sampling is performed on the first image data based on the matrix to obtain matrix data;
[0025] The first scaling factor configuration and the second scaling factor configuration are multiplied by the matrix data respectively to obtain the first data configuration and the second data configuration, and the first filter component and the second filter component are generated according to the first data configuration and the second data configuration respectively.
[0026] The first image data and the first filtered component are differentially calculated to obtain the sharpened image data;
[0027] The sharpened image data and the second filtered component are superimposed to obtain the target image data.
[0028] This preferred solution uses a matrix to sample pixels from the first image data, accurately acquiring pixel information and providing a precise data foundation for subsequent processing. By performing differential calculations on the first image data and the first filtered component, sharpened image data is obtained, enhancing image edges and details. Superimposing the sharpened image data with the second filtered component to obtain the target image data achieves image smoothing, reducing noise and distortion.
[0029] As a preferred embodiment, the first scaling factor configuration and the second scaling factor configuration are calculated in the sharpening module and the smoothing module, respectively.
[0030] If the individual enhancement effect configured by the first ratio coefficient corresponding to the sharpening module does not meet the preset standard, the individual enhancement effect of the sharpening module is improved by configuring the center point coefficient value of the smoothing module to be less than one; the individual enhancement effect is determined according to the blur and sharpness of the image quality.
[0031] In this preferred scheme, the coordinated adjustment of the center point coefficient values between the sharpening and smoothing modules facilitates efficient collaboration between the two modules, improving overall processing efficiency. This synergy allows for more efficient resource utilization. Compared to directly increasing the processing intensity of the sharpening module or introducing new processing algorithms, improving the smoothing effect by adjusting the center point coefficient values of the smoothing module achieves the optimization goal without significantly increasing the computational burden.
[0032] As a preferred embodiment, the original image data is scaled to obtain the first image data, specifically as follows:
[0033] The original image data is converted into parallel data, and the converted original image data is reduced based on the reduction module to obtain reduced image data;
[0034] The reduced image data is cached in the image caching module;
[0035] The control magnification module obtains the reduced image data from the image cache module and magnifies the reduced image data to obtain the first image data; wherein, the reduction module, the image cache module and the magnification module are all devices in the FPGA system.
[0036] This preferred solution employs a scaling mode of "shrinking + caching + enlarging," ensuring that the maximum resolution of the image data entering the corresponding memory cache does not exceed that of the original image. This method sets a theoretical upper limit on bandwidth usage, helping to avoid the need for the memory to cache more data than the original image during the enlargement process, thereby significantly reducing the bandwidth requirements of the memory.
[0037] This application also provides a scaling image optimization device based on a matrix system, including a data module, a scaling module, and a synthesis module;
[0038] The data module is used to acquire the raw image data of the video input.
[0039] The scaling module is used to scale the original image data to obtain the first image data;
[0040] The integrated module is used to sequentially sharpen and smooth the first image data based on a first proportional coefficient configuration and a second proportional coefficient configuration to obtain target image data; wherein, the first proportional coefficient configuration is established by maintaining a constant edge coefficient ratio with the center point coefficient value as the primary factor; the second proportional coefficient configuration is established by maintaining a balanced distribution of edge coefficients with the center point coefficient value as the primary factor.
[0041] As a preferred embodiment, the integrated module includes a central unit, a coefficient unit, a partitioning unit, and a configuration unit;
[0042] The central unit is used to obtain the center point coefficient value sent by the client;
[0043] The coefficient unit is used to take the center point coefficient value as the center point coefficient of the sharpening filter matrix, and calculate the difference between the center point coefficient value and the preset maximum value to obtain the coefficient space;
[0044] The partitioning unit is used to partition the coefficient space in the sharpening filter matrix by controlling the diagonal neighbor points and the four neighbor points according to a preset ratio, so as to obtain the edge coefficient configuration of the sharpening filter matrix.
[0045] The configuration unit is configured to form the first scaling factor configuration for sharpening processing by the center point coefficient value and the edge coefficient configuration of the sharpening filter matrix.
[0046] As a preferred embodiment, the integrated module includes an acquisition unit, a difference unit, an equalization unit, and a construction unit;
[0047] The acquisition unit is used to acquire the center point coefficient value sent by the client.
[0048] The difference unit is used to take the center point coefficient value as the center point coefficient of the smoothing filter matrix, and calculate the difference between the center point coefficient value and the preset maximum value of the coefficient to obtain the coefficient space.
[0049] The equal division unit is used to control all neighborhood edge points to equally divide the coefficient space in the smoothing filter matrix according to a preset ratio, so as to obtain the edge coefficient configuration of the smoothing filter matrix.
[0050] The construction unit is configured to form the second scaling factor configuration for smoothing processing by the center point coefficient values and the edge coefficient configuration of the smoothing filter matrix.
[0051] As a preferred embodiment, the integrated module includes a sampling unit, a processing unit, a sharpening unit, and a smoothing unit;
[0052] The sampling unit is used to perform pixel sampling on the first image data based on a matrix to obtain matrix data;
[0053] The processing unit is configured to multiply the first scaling factor configuration and the second scaling factor configuration by the matrix data respectively to obtain a first data configuration and a second data configuration, and generate a first filter component and a second filter component according to the first data configuration and the second data configuration respectively.
[0054] The sharpening unit is used to perform differential calculation on the first image data and the first filtered component to obtain sharpened image data.
[0055] The smoothing unit is used to perform superposition calculation on the sharpened image data and the second filtered component to obtain the target image data.
[0056] As a preferred embodiment, the first scaling factor configuration and the second scaling factor configuration are calculated in the sharpening module and the smoothing module, respectively.
[0057] If the individual enhancement effect configured by the first ratio coefficient corresponding to the sharpening module does not meet the preset standard, the individual enhancement effect of the sharpening module is improved by configuring the center point coefficient value of the smoothing module to be less than one; the individual enhancement effect is determined according to the blur and sharpness of the image quality.
[0058] As a preferred embodiment, the scaling module includes a shrinking unit, a buffering unit, and a zooming unit;
[0059] The reduction unit is used to convert the original image data into parallel data, and based on the reduction module, to perform reduction processing on the converted original image data to obtain reduced image data;
[0060] The caching unit is used to cache the reduced image data into the image caching module;
[0061] The magnification unit is used to control the magnification module to obtain the reduced image data from the image cache module, and to magnify the reduced image data to obtain the first image data; wherein, the reduction module, the image cache module and the magnification module are all devices in the FPGA system.
[0062] This application also provides a storage medium storing a computer program, which is called and executed by a computer to implement the matrix system-based image scaling optimization method described above. Attached Figure Description
[0063] Figure 1 This is a flowchart illustrating a scaling image optimization method based on a matrix system provided in an embodiment of this application;
[0064] Figure 2 This is an external structure diagram provided in the embodiments of this application;
[0065] Figure 3 This is a structural diagram of the image optimization module provided in an embodiment of this application;
[0066] Figure 4 This is a flowchart of the FPGA system provided in the embodiments of this application;
[0067] Figure 5 This is a schematic diagram of a scaling image optimization device based on a matrix system provided in an embodiment of this application. Detailed Implementation
[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0069] In the description of this application, it should be understood that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," and "third" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "several" means two or more.
[0070] The scaling image optimization method based on a matrix system provided in this application is mainly applied to situations where it is necessary to optimize the performance of the FPGA output screen, optimize and balance the blurred or sharp image after scaling / the original image, thereby improving the quality of the final output image.
[0071] Example 1:
[0072] Please see Figure 1 The embodiments of this application provide a scaling image optimization method based on a matrix system, including S1~S3, and the specific implementation steps are as follows:
[0073] S1. Obtain the raw image data of the video input.
[0074] Step S1 in this embodiment of the application is specifically as follows:
[0075] Acquire the raw image data of the video input; the raw image data of the video input can be captured using multimedia frameworks provided by the operating system (such as DirectShow, V4L2, Media Foundation, etc.) or third-party libraries (such as OpenCV, FFmpeg, etc.).
[0076] For examples of this application, please refer to [link / reference]. Figure 2-3 ;
[0077] Figure 2 This is an external structure diagram provided in the embodiments of this application. The core focus of this application is the development and design of the FPGA side, and its overall peripheral architecture is as follows: Figure 2 As shown. It should be noted that this application focuses on the field of image optimization in matrix systems, and the subsequent discussion will closely revolve around this core theme of image optimization. Other non-core functions will not be elaborated upon. In terms of the external structure of the application, its design is simple and clear, mainly composed of two major components: MCU (Microcontroller Unit) and FPGA (Field Programmable Gate Array) chip; among which, the FPGA chip is responsible for performing image data scaling and image optimization tasks, while the MCU is responsible for sending the required configuration parameters to the FPGA. These parameters are pre-configured by the client and transmitted to the FPGA for caching via the MCU.
[0078] Figure 3 This is a structural diagram of the image optimization module provided in the embodiments of this application, which shows the structural framework of the image optimization algorithm in this application. The implementation scheme will be explained and described in detail below with reference to this structural diagram.
[0079] S2. Scale the original image data to obtain the first image data.
[0080] Step S2 in this embodiment of the application is specifically as follows:
[0081] The original image data is converted from serial to parallel data by LVDS. Based on the downsizing module, the single frame image in the converted original image data is downsized to obtain downsized image data.
[0082] Cache single-frame images from the scaled-down image data into the image cache module;
[0083] The control magnification module obtains the reduced image data from the image cache module and magnifies the reduced image data to obtain the first image data; wherein, the reduction module, the image cache module and the magnification module are all devices in the FPGA system, and the FPGA system also includes devices such as the sharpening module and the smoothing module.
[0084] An "enhancement factor" is introduced, which is adjusted according to the change between the resolution of the input image and the resolution of the scaled output image. This enhancement factor increases or decreases accordingly when the image is enlarged or reduced to ensure that the optimization effect matches the degree of image scaling, thereby achieving the best image quality performance for the initial image data.
[0085] In embodiment S2 of this application, a scaling mode of "shrinking + caching + enlarging" is chosen instead of "scaling + caching" because the latter has significant design flaws. Specifically, the "scaling + caching" mode, when enlarging an image, causes the amount of image data that needs to be cached within the DDR (Double Data Rate Synchronous Dynamic Random-Access Memory) to far exceed the original image, thus placing extremely high demands on the DDR bandwidth. Furthermore, higher resolution means a higher clock frequency is required to perform clock domain switching on the data in the DDR, which undoubtedly increases the difficulty of timing convergence for the FPGA device. In contrast, the scaling mode adopted in this application ensures that the maximum resolution of the image does not exceed the original size before the data enters the DDR cache, thereby setting a theoretical upper limit on bandwidth usage. As long as this upper limit is kept within the actual bandwidth of the DDR, the stable operation of the system can be effectively guaranteed, helping to avoid the need for the memory to cache more data than the original image during the enlargement process, thus significantly reducing the bandwidth requirements of the memory.
[0086] S3. Based on the first scaling factor configuration and the second scaling factor configuration, the first image data is sharpened and smoothed sequentially to obtain the target image data; wherein, the first scaling factor configuration is based on the center point coefficient value and is established by keeping the edge coefficient ratio constant; the second scaling factor configuration is based on the center point coefficient value and is established by keeping the edge coefficient evenly distributed.
[0087] Step S3 in this embodiment includes S3.1 to S3.3, wherein S3.1 is the process of establishing the first scaling factor configuration for sharpening, S3.2 is the process of establishing the second scaling factor configuration for smoothing, and S3.3 is the process of sharpening and smoothing the image data, specifically as follows:
[0088] S3.1 Obtain the center point coefficient value sent by the client; the center point coefficient value needs to be sent by the client via MCU configuration; it should be noted that before inputting it into the matrix operation, the coefficient value needs to be finely adjusted according to the actual scaling degree. Since this coefficient value has a decisive influence on the actual effect of subsequent filtering processing, its value must be strictly controlled to ensure that it can adapt to different scaling requirements, thereby ensuring that the result of the optimization calculation is neither too large nor too small, but maintained within a reasonable range.
[0089] The center point coefficient values are used as the center point coefficients of the sharpening filter matrix, and the difference between the center point coefficient values and the preset maximum values is calculated to obtain the coefficient space;
[0090] In the sharpening filter matrix, the diagonal neighbor points (diagonal points) and four neighbor points (top, bottom, left, and right points) are used to divide the coefficient space according to a preset ratio to obtain the edge coefficient configuration of the sharpening filter matrix; where the preset ratio is: "diagonal neighbor points: four neighbor points = 1:2 relationship";
[0091] The first scaling factor configuration for sharpening is formed by the center point coefficient values and edge coefficient configuration of the sharpening filter matrix.
[0092] In this embodiment S3.1, the client can dynamically adjust the center point coefficient value according to different image content, quality requirements, or application scenarios to adapt to changing processing needs. The coefficient space is divided according to a preset ratio by controlling the diagonal and quadruple neighbor points, making the edge coefficient configuration more detailed and precise. Furthermore, this detailed edge coefficient configuration helps enhance the edge effect of the image, making the image appear clearer and sharper.
[0093] S3.2. Use the center point coefficient values as the center point coefficients of the smoothing filter matrix, and calculate the difference between the center point coefficient values and the preset maximum coefficient values to obtain the coefficient space;
[0094] In the smoothing filter matrix, the coefficient space is divided equally by all neighborhood edge points according to a preset ratio to obtain the edge coefficient configuration of the smoothing filter matrix; wherein, the proportion of the center point in the new value is 1, and the proportion of the neighborhood edge points is 1 / 8.
[0095] The second scaling factor configuration for smoothing is formed by the center point coefficient values and edge coefficient configurations of the smoothing filter matrix.
[0096] The first and second proportional coefficient configurations are calculated in the sharpening and smoothing modules, respectively. The smoothing and sharpening modules correspond to low-pass filtering and high-pass filtering, respectively. The overall architecture of the sharpening and smoothing modules is basically similar, with the main difference being the internal matrix algorithm.
[0097] If the individual enhancement effect of the first proportional coefficient configured for the sharpening module does not meet the preset standard, the individual enhancement effect of the sharpening module is improved by configuring the center point coefficient value of the smoothing module to be less than one; conversely, the individual enhancement effect of the smoothing module is improved by configuring the center point coefficient value of the sharpening module to be less than one.
[0098] Furthermore, the individual enhancement effect is determined based on the degree of blurriness and sharpness in image quality. Specifically, even after targeted optimization of a single module and maximizing its optimization capabilities, the output image may still exhibit noticeable blurriness or sharpness. In image presentation, a key factor influencing the human eye's perception lies in the differences in pixel values at image edges. Specifically, sharp image edges are characterized by significant features and large numerical differences between adjacent pixels, resulting in a clearer image perceived by the human eye. Conversely, smooth image edges exhibit smaller differences between adjacent pixels, leading to a more blurred perception of the image.
[0099] It should be noted that in the field of image processing, using FPGAs to implement smoothing and sharpening filters is a common technique. Traditional methods typically involve directly multiplying the image matrix data with preset filter template coefficients to calculate the filtered components. Specifically, smoothing algorithms add edge points to the center point after multiplication, while sharpening algorithms subtract the sum of edge points from the center point. However, this method has a significant drawback: the filter coefficients remain fixed, resulting in an uncontrollable and constant degree of smoothing or sharpening in the final output image.
[0100] Taking a 3×3 matrix in the sharpening module as an example, in the traditional approach, the coefficient of the center point in the new value is fixed at 1 / 4, the coefficients of the four points above, below, left, and right are 1 / 8, and the coefficients of the four diagonal points are 1 / 16. The sum of the overall coefficients is strictly equal to 1, that is, (1×1 / 4)+(4×1 / 8)+(4×1 / 16)=1. This fixed coefficient allocation method limits the flexibility of the filtering intensity and cannot meet the needs of dynamic adjustment of image smoothing or sharpening in practical applications.
[0101] In this embodiment, steps S3.1 and S3.2 configure the overall filtering intensity using center point coefficient values. By changing the center point coefficient values, the filtering intensity can be flexibly adjusted to adapt to different image processing needs. This dynamic adjustment capability makes the processing algorithm more flexible and versatile; furthermore, different images may require different degrees of sharpening or smoothing. Configuring the filtering intensity using center point coefficient values allows for personalized processing based on the characteristics of different images, achieving better visual results.
[0102] Example S3.2 controls all neighborhood edge points to divide the coefficient space equally according to a preset ratio, so that the configuration of edge coefficients is more balanced and stable; and the balanced edge coefficient configuration helps to reduce distortion in the image processing process, making the image look more natural and smooth.
[0103] Furthermore, the coordinated adjustment of the center point coefficient values between the sharpening and smoothing modules facilitates efficient collaboration between the two modules, improving overall processing efficiency. This synergy allows for more efficient resource utilization. Compared to directly increasing the processing intensity of the sharpening module or introducing new processing algorithms, improving the smoothing effect by adjusting the center point coefficient values of the smoothing module can achieve the optimization goal without significantly increasing the computational burden.
[0104] S3.3. Using a 3×3 matrix, each pixel in the first image data and its eight neighboring pixels are sampled to obtain matrix data. Specifically, the first image data is fed into a first FIFO module and a second FIFO module for buffering. The first FIFO module directly receives the original first image data, while the second FIFO module receives the output data from the first FIFO module. Thus, by combining the data buffered by the first FIFO module, the data buffered by the second FIFO module, and the original first image data, three rows of data can be output simultaneously. These three rows of data are then fed into a 3×3 matrix register for further buffering.
[0105] Based on the first, second, and third row data multipliers, the first scaling factor configuration and the second scaling factor configuration are multiplied by the matrix data respectively to obtain the first data configuration and the second data configuration;
[0106] The first and second data configurations are respectively input into the filtering operation module for filtering operations to calculate the filtered components, i.e., the first filtered component and the second filtered component. The first filtered component and the second filtered component are then input into the calculation module. The first filtered component required for sharpening is obtained by calculating the difference between the center point pixel value (center point coefficient value) and the edge point pixel value (edge coefficient configuration) and taking the absolute value of the difference. The second filtered component required for smoothing is obtained by adding the edge point pixel value and the center point pixel value.
[0107] In the calculation module, the first filter component is subtracted from the first image data to obtain the sharpened image data; the sharpened image data and the second filter component are added to obtain the target image data, i.e., the optimized data.
[0108] The output switching mechanism of the output selection module allows users to choose between outputting the target image data or the original first image data as the final image data. The instructions for the output switching mechanism are set according to the configuration issued by the client.
[0109] For examples of this application, please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart of an FPGA system provided in an embodiment of this application, illustrating the data processing flow and related operation methods within the FPGA of this application. The processing flow is roughly as follows:
[0110] The video input image data is first processed by LVDS serial-to-parallel conversion, and then sent to the downsizing module for downsizing of single frames.
[0111] The scaled-down image frames are stored in the image cache module, which typically uses DDR3 technology to achieve efficient data caching;
[0112] Then, the magnification module extracts data from the image cache module and performs the corresponding magnification processing;
[0113] After the algorithm processing is completed, the data is passed to the sharpening module, which aims to reduce overly sharp pixel values in the image to achieve a smoothing effect.
[0114] The smoothed data then enters the smoothing module, which enhances the clarity of the blurred image by increasing pixel interpolation at the image edges.
[0115] In this embodiment, S3.3 performs pixel sampling on the first image data using a matrix, which can accurately obtain the pixel information of the image, providing an accurate data foundation for subsequent processing. By performing differential calculation on the first image data and the first filtered component, the sharpened image data can be obtained, enhancing the edges and details of the image. By superimposing the sharpened image data and the second filtered component to obtain the target image data, image smoothing can be achieved, reducing noise and distortion.
[0116] Overall, this embodiment has the following beneficial effects:
[0117] In this invention, scaling can adjust the image size as needed to adapt to different display devices or application scenarios. Furthermore, it can reduce the computational load required for subsequent processing steps while maintaining image quality, thus improving overall processing efficiency. Based on a first scaling factor configuration and a second scaling factor configuration, the first image data is sharpened and smoothed, respectively. This is a key step in improving image quality. The first scaling factor configuration prioritizes the center point coefficient value, meaning that when processing the image, more attention is paid to information in the central area of the image. This helps preserve the main features and details of the image, resulting in a clearer image. Moreover, by maintaining a constant edge coefficient ratio, the first scaling factor configuration ensures consistency when processing image edges, helping to reduce distortion and blurring at edges and improve overall image quality. The second scaling factor configuration also prioritizes the center point coefficient value, but it is established by maintaining a balanced distribution of edge coefficients. This configuration may help reduce noise and details in the image, making the image smoother and softer. This helps improve the quality of the final output image, meeting different application scenarios and needs.
[0118] In summary, this application demonstrates significant advantages in improving image quality after scaling, and also significantly enhances the detail representation of blurred image edges while moderately softening overly sharp image edges, thereby achieving a balanced and optimized overall image effect. Most importantly, compared to the limitations of traditional FPGA filtering schemes in controlling the back-end filtering intensity, this application innovatively introduces a method to precisely control the overall filtering intensity by adjusting the center point coefficient value. This improvement provides users with more delicate and flexible image processing capabilities, greatly enriching the precision and operability of image processing.
[0119] Example 2:
[0120] Please see Figure 5 The embodiments of this application provide a scaling image optimization device based on a matrix system, including a data module 10, a scaling module 20 and a synthesis module 30;
[0121] Among them, the data module 10 is used to acquire the raw image data of the video input;
[0122] Scaling module 20 is used to scale the original image data to obtain the first image data;
[0123] The integration module 30 is used to perform sharpening and smoothing processing on the first image data sequentially based on the first scaling factor configuration and the second scaling factor configuration to obtain the target image data; wherein, the first scaling factor configuration is established by maintaining the edge coefficient ratio constant, with the center point coefficient value as the main factor; the second scaling factor configuration is established by maintaining the edge coefficient distribution evenly, with the center point coefficient value as the main factor.
[0124] In one embodiment, data module 10 specifically comprises:
[0125] Acquire the raw image data of the video input; the raw image data of the video input can be captured using multimedia frameworks provided by the operating system (such as DirectShow, V4L2, Media Foundation, etc.) or third-party libraries (such as OpenCV, FFmpeg, etc.).
[0126] For examples of this application, please refer to [link / reference]. Figure 2-3 ;
[0127] Figure 2 This is an external structure diagram provided in the embodiments of this application. The core focus of this application is the development and design of the FPGA side, and its overall peripheral architecture is as follows: Figure 2 As shown. It should be noted that this application focuses on the field of image optimization in matrix systems, and the subsequent discussion will closely revolve around this core theme of image optimization. Other non-core functions will not be elaborated upon. In terms of the external structure of the application, its design is simple and clear, mainly composed of two major components: MCU (Microcontroller Unit) and FPGA (Field Programmable Gate Array) chip; among which, the FPGA chip is responsible for performing image data scaling and image optimization tasks, while the MCU is responsible for sending the required configuration parameters to the FPGA. These parameters are pre-configured by the client and transmitted to the FPGA for caching via the MCU.
[0128] Figure 3 This is a structural diagram of the image optimization module provided in the embodiments of this application, which shows the structural framework of the image optimization algorithm in this application. The implementation scheme will be explained and described in detail below with reference to this structural diagram.
[0129] In one embodiment, the scaling module 20 includes a shrinking unit, a caching unit, a zooming unit, and an optimization unit;
[0130] The reduction unit is used to convert the original image data from serial to parallel data via LVDS, and based on the reduction module, to reduce the size of a single frame in the converted original image data to obtain reduced image data.
[0131] A caching unit is used to cache single-frame images from the scaled-down image data into the image caching module;
[0132] The magnification unit is used to control the magnification module to obtain reduced image data from the image cache module and to magnify the reduced image data to obtain the first image data. The reduction module, the image cache module and the magnification module are all devices in the FPGA system. The FPGA system also includes devices such as the sharpening module and the smoothing module.
[0133] The optimization unit introduces an "enhancement factor," which is adjusted according to the change between the resolution of the input image and the resolution of the scaled output image. This enhancement factor increases or decreases accordingly when the image is enlarged or reduced to ensure that the optimization effect matches the degree of image scaling, thereby achieving optimal image quality for the initial image data.
[0134] In embodiment S2 of this application, a scaling mode of "shrinking + caching + enlarging" is chosen instead of "scaling + caching" because the latter has significant design flaws. Specifically, the "scaling + caching" mode, when enlarging an image, causes the amount of image data that needs to be cached within the DDR (Double Data Rate Synchronous Dynamic Random-Access Memory) to far exceed the original image, thus placing extremely high demands on the DDR bandwidth. Furthermore, higher resolution means a higher clock frequency is required to perform clock domain switching on the data in the DDR, which undoubtedly increases the difficulty of timing convergence for the FPGA device. In contrast, the scaling mode adopted in this application ensures that the maximum resolution of the image does not exceed the original size before the data enters the DDR cache, thereby setting a theoretical upper limit on bandwidth usage. As long as this upper limit is kept within the actual bandwidth of the DDR, the stable operation of the system can be effectively guaranteed, helping to avoid the need for the memory to cache more data than the original image during the enlargement process, thus significantly reducing the bandwidth requirements of the memory.
[0135] In one embodiment, the integration module 30 includes a center unit, a coefficient unit, a division unit, a configuration unit, a difference unit, a averaging unit, a construction unit, a sampling unit, a processing unit, a sharpening-smoothing unit, and an output unit; wherein, the center unit, coefficient unit, division unit, and configuration unit are the process of establishing the first proportional coefficient configuration for sharpening processing, the difference unit, averaging unit, and construction unit are the process of establishing the second proportional coefficient configuration for smoothing processing, and the sampling unit, processing unit, sharpening-smoothing unit, and output unit are the process of sharpening and smoothing image data;
[0136] The central unit is used to obtain the center point coefficient value sent by the client. The center point coefficient value needs to be sent by the client via the MCU. It should be noted that before inputting it into the matrix operation, the coefficient value needs to be finely adjusted according to the actual scaling degree. Since this coefficient value has a decisive influence on the actual effect of subsequent filtering processing, its value must be strictly controlled to ensure that it can adapt to different scaling requirements, thereby ensuring that the result of the optimization calculation is neither too large nor too small, but maintained within a reasonable range.
[0137] The coefficient unit is used to take the center point coefficient value as the center point coefficient of the sharpening filter matrix, and calculate the difference between the center point coefficient value and the preset maximum value to obtain the coefficient space;
[0138] The partitioning unit is used to divide the coefficient space in the sharpening filter matrix according to a preset ratio by controlling the diagonal neighbor points (diagonal points) and the four neighbor points (top, bottom, left, and right points) to obtain the edge coefficient configuration of the sharpening filter matrix; wherein, the preset ratio is: "diagonal neighbor points: four neighbor points = 1:2 relationship";
[0139] A configuration unit is used to form the first scaling factor configuration for sharpening processing, which consists of the center point coefficient values and edge coefficient configurations of the sharpening filter matrix.
[0140] In this embodiment, within the center unit, coefficient unit, partitioning unit, and configuration unit, the client can dynamically adjust the center point coefficient value according to different image content, quality requirements, or application scenarios to adapt to varying processing needs. The coefficient space is partitioned according to a preset ratio using diagonal and quadruple neighbor points, resulting in more detailed and precise edge coefficient configuration. Furthermore, this detailed edge coefficient configuration helps enhance the image's edge effects, making the image appear clearer and sharper.
[0141] The difference unit is used to take the center point coefficient value as the center point coefficient of the smoothing filter matrix, and calculate the difference between the center point coefficient value and the preset maximum value of the coefficient to obtain the coefficient space;
[0142] The equal division unit is used to control all neighborhood edge points in the smoothing filter matrix to divide the coefficient space equally according to a preset ratio, so as to obtain the edge coefficient configuration of the smoothing filter matrix; wherein, the proportion of the center point in the new value is 1, and the proportion of the neighborhood edge points is 1 / 8.
[0143] The equal-division unit is used to form a second proportional coefficient configuration for smoothing, consisting of the center point coefficient values and edge coefficient configurations of the smoothing filter matrix.
[0144] The first and second proportional coefficient configurations are calculated in the sharpening and smoothing modules, respectively. The smoothing and sharpening modules correspond to low-pass filtering and high-pass filtering, respectively. The overall architecture of the sharpening and smoothing modules is basically similar, with the main difference being the internal matrix algorithm.
[0145] If the individual enhancement effect of the first proportional coefficient configured for the sharpening module does not meet the preset standard, the individual enhancement effect of the sharpening module is improved by configuring the center point coefficient value of the smoothing module to be less than one; conversely, the individual enhancement effect of the smoothing module is improved by configuring the center point coefficient value of the sharpening module to be less than one.
[0146] Furthermore, the individual enhancement effect is determined based on the degree of blurriness and sharpness in image quality. Specifically, even after targeted optimization of a single module and maximizing its optimization capabilities, the output image may still exhibit noticeable blurriness or sharpness. In image presentation, a key factor influencing the human eye's perception lies in the differences in pixel values at image edges. Specifically, sharp image edges are characterized by significant features and large numerical differences between adjacent pixels, resulting in a clearer image perceived by the human eye. Conversely, smooth image edges exhibit smaller differences between adjacent pixels, leading to a more blurred perception of the image.
[0147] It should be noted that in the field of image processing, using FPGAs to implement smoothing and sharpening filters is a common technique. Traditional methods typically involve directly multiplying the image matrix data with preset filter template coefficients to calculate the filtered components. Specifically, smoothing algorithms add edge points to the center point after multiplication, while sharpening algorithms subtract the sum of edge points from the center point. However, this method has a significant drawback: the filter coefficients remain fixed, resulting in an uncontrollable and constant degree of smoothing or sharpening in the final output image.
[0148] Taking a 3×3 matrix in the sharpening module as an example, in the traditional approach, the coefficient of the center point in the new value is fixed at 1 / 4, the coefficients of the four points above, below, left, and right are 1 / 8, and the coefficients of the four diagonal points are 1 / 16. The sum of the overall coefficients is strictly equal to 1, that is, (1×1 / 4)+(4×1 / 8)+(4×1 / 16)=1. This fixed coefficient allocation method limits the flexibility of the filtering intensity and cannot meet the needs of dynamic adjustment of image smoothing or sharpening in practical applications.
[0149] In this embodiment, the center unit to the construction unit configures the overall filtering intensity using center point coefficient values. By changing the center point coefficient values, the filtering intensity can be flexibly adjusted to adapt to different image processing needs. This dynamic adjustment capability makes the processing algorithm more flexible and versatile; furthermore, different images may require different degrees of sharpening or smoothing. Configuring the filtering intensity using center point coefficient values allows for personalized processing based on the characteristics of different images, achieving better visual results.
[0150] In the embodiment, the difference unit, the equalization unit, and the construction unit control all neighborhood edge points to divide the coefficient space equally according to a preset ratio, so that the configuration of edge coefficients is more balanced and stable; and the balanced configuration of edge coefficients helps to reduce distortion in the image processing process, making the image look more natural and smooth.
[0151] Furthermore, the coordinated adjustment of the center point coefficient values between the sharpening and smoothing modules facilitates efficient collaboration between the two modules, improving overall processing efficiency. This synergy allows for more efficient resource utilization. Compared to directly increasing the processing intensity of the sharpening module or introducing new processing algorithms, improving the smoothing effect by adjusting the center point coefficient values of the smoothing module can achieve the optimization goal without significantly increasing the computational burden.
[0152] The sampling unit is used to sample each pixel and its eight neighboring pixels in the first image data using a 3×3 matrix pattern to obtain matrix data. Specifically, the first image data is fed into a first FIFO module and a second FIFO module for buffering. The first FIFO module directly receives the original first image data, while the second FIFO module receives the output data from the first FIFO module. By combining the data buffered by the first FIFO module, the data buffered by the second FIFO module, and the original first image data, three rows of data can be output simultaneously. These three rows of data are then fed into a 3×3 matrix register for further buffering.
[0153] The processing unit is used to multiply the first scaling factor configuration and the second scaling factor configuration with the matrix data respectively based on the first, second and third row data multipliers to obtain the first data configuration and the second data configuration.
[0154] The processing unit is further configured to input the first data configuration and the second data configuration into the filtering operation module for filtering operations, thereby calculating the filtered components, i.e., obtaining the first filtered component and the second filtered component, and input the first filtered component and the second filtered component into the calculation module; wherein, the first filtered component required for sharpening processing is obtained by calculating the difference between the center point pixel value (center point coefficient value) and the edge point pixel value (edge coefficient configuration), and taking the absolute value of the difference; while the second filtered component required for smoothing processing is obtained by adding the edge point pixel value and the center point pixel value.
[0155] The sharpening-smoothing unit is used in the calculation module to subtract the first filter component from the first image data to obtain the sharpened image data; and to add the sharpened image data and the second filter component to obtain the target image data, i.e., the optimized data.
[0156] The output unit is used to select either the target image data or the original first image data as the final image data output through the output switching mechanism of the output selection module; wherein, the instruction of the output switching mechanism is set according to the configuration issued by the client.
[0157] For examples of this application, please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart of an FPGA system provided in an embodiment of this application, illustrating the data processing flow and related operation methods within the FPGA of this application. The processing flow is roughly as follows:
[0158] The video input image data is first processed by LVDS serial-to-parallel conversion, and then sent to the downsizing module for downsizing of single frames.
[0159] The scaled-down image frames are stored in the image cache module, which typically uses DDR3 technology to achieve efficient data caching;
[0160] Then, the magnification module extracts data from the image cache module and performs the corresponding magnification processing;
[0161] After the algorithm processing is completed, the data is passed to the sharpening module, which aims to reduce overly sharp pixel values in the image to achieve a smoothing effect.
[0162] The smoothed data then enters the smoothing module, which enhances the clarity of the blurred image by increasing pixel interpolation at the image edges.
[0163] In this embodiment, the sampling unit, processing unit, sharpening-smoothing unit, and output unit perform pixel sampling on the first image data using a matrix, which can accurately obtain the pixel information of the image and provide an accurate data foundation for subsequent processing. By performing differential calculation on the first image data and the first filtered component, the sharpened image data can be obtained, enhancing the edges and details of the image. By superimposing the sharpened image data and the second filtered component to obtain the target image data, image smoothing can be achieved, reducing noise and distortion.
[0164] Overall, this embodiment has the following beneficial effects:
[0165] In this invention, scaling can adjust the image size as needed to adapt to different display devices or application scenarios. Furthermore, it can reduce the computational load required for subsequent processing steps while maintaining image quality, thus improving overall processing efficiency. Based on a first scaling factor configuration and a second scaling factor configuration, the first image data is sharpened and smoothed, respectively. This is a key step in improving image quality. The first scaling factor configuration prioritizes the center point coefficient value, meaning that when processing the image, more attention is paid to information in the central area of the image. This helps preserve the main features and details of the image, resulting in a clearer image. Moreover, by maintaining a constant edge coefficient ratio, the first scaling factor configuration ensures consistency when processing image edges, helping to reduce distortion and blurring at edges and improve overall image quality. The second scaling factor configuration also prioritizes the center point coefficient value, but it is established by maintaining a balanced distribution of edge coefficients. This configuration may help reduce noise and details in the image, making the image smoother and softer. This helps improve the quality of the final output image, meeting different application scenarios and needs.
[0166] In summary, this application demonstrates significant advantages in improving image quality after scaling, and also significantly enhances the detail representation of blurred image edges while moderately softening overly sharp image edges, thereby achieving a balanced and optimized overall image effect. Most importantly, compared to the limitations of traditional FPGA filtering schemes in controlling the back-end filtering intensity, this application innovatively introduces a method to precisely control the overall filtering intensity by adjusting the center point coefficient value. This improvement provides users with more delicate and flexible image processing capabilities, greatly enriching the precision and operability of image processing.
[0167] Example 3:
[0168] This application provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the aforementioned matrix system-based image scaling optimization method.
[0169] The image scaling optimization method based on a matrix system, when implemented as a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0170] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A scaling image optimization method based on a matrix system, characterized in that, include: Obtain the raw image data of the video input; The original image data is scaled to obtain the first image data; Based on the first and second proportional coefficient configurations, the first image data is sequentially sharpened and smoothed to obtain target image data. The second proportional coefficient configuration is established by maintaining a balanced distribution of edge coefficients, with the center point coefficient value as the primary factor. The first proportional coefficient configuration is established by maintaining a constant edge coefficient ratio, specifically: obtaining the center point coefficient value sent by the client; using the center point coefficient value as the center point coefficient of the sharpening filter matrix, and calculating the difference between the center point coefficient value and a preset maximum value to obtain the coefficient space; dividing the coefficient space in the sharpening filter matrix according to a preset ratio using diagonal and quadruple neighbor points to obtain the edge coefficient configuration of the sharpening filter matrix; the center point coefficient value and the edge coefficient configuration of the sharpening filter matrix constitute the first proportional coefficient configuration for sharpening processing.
2. The image scaling optimization method based on a matrix system as described in claim 1, characterized in that, The second proportional coefficient configuration is based on the center point coefficient value and is established by maintaining a balanced distribution of edge coefficients, specifically as follows: Get the center point coefficient value sent by the client; The center point coefficient values are used as the center point coefficients of the smoothing filter matrix, and the difference between the center point coefficient values and the preset maximum coefficient values is calculated to obtain the coefficient space. In the smoothing filter matrix, the coefficient space is divided equally by controlling all neighborhood edge points according to a preset ratio to obtain the edge coefficient configuration of the smoothing filter matrix; The second scaling factor configuration for smoothing is constituted by the center point coefficient values and the edge coefficient configuration of the smoothing filter matrix.
3. The image scaling optimization method based on a matrix system as described in claim 1, characterized in that, Based on the first scaling factor configuration and the second scaling factor configuration, the first image data is sequentially sharpened and smoothed to obtain the target image data, specifically as follows: Pixel sampling is performed on the first image data based on the matrix to obtain matrix data; The first scaling factor configuration and the second scaling factor configuration are multiplied by the matrix data respectively to obtain the first data configuration and the second data configuration, and the first filter component and the second filter component are generated according to the first data configuration and the second data configuration respectively. The first image data and the first filtered component are differentially calculated to obtain the sharpened image data; The sharpened image data and the second filtered component are superimposed to obtain the target image data.
4. The image scaling optimization method based on a matrix system as described in claim 1, characterized in that, The first scaling factor configuration and the second scaling factor configuration are calculated in the sharpening module and the smoothing module, respectively. If the individual enhancement effect configured by the first ratio coefficient corresponding to the sharpening module does not meet the preset standard, the individual enhancement effect of the sharpening module is improved by configuring the center point coefficient value of the smoothing module to be less than one; the individual enhancement effect is determined according to the blur and sharpness of the image quality.
5. The image scaling optimization method based on a matrix system as described in claim 1, characterized in that, The original image data is scaled to obtain the first image data, specifically as follows: The original image data is converted into parallel data, and the converted original image data is reduced based on the reduction module to obtain reduced image data; The reduced image data is cached in the image caching module; The control magnification module obtains the reduced image data from the image cache module and magnifies the reduced image data to obtain the first image data; wherein, the reduction module, the image cache module and the magnification module are all devices in the FPGA system.
6. A scaling image optimization device based on a matrix system, characterized in that, It includes a data module, a scaling module, and a comprehensive module; The data module is used to acquire the raw image data of the video input. The scaling module is used to scale the original image data to obtain the first image data; The integrated module is used to sequentially sharpen and smooth the first image data based on a first proportional coefficient configuration and a second proportional coefficient configuration to obtain target image data. The second proportional coefficient configuration is established by maintaining a balanced distribution of edge coefficients, with the center point coefficient value as the primary factor. The first proportional coefficient configuration is established by maintaining a constant edge coefficient ratio, specifically: obtaining the center point coefficient value sent by the client; using the center point coefficient value as the center point coefficient of the sharpening filter matrix, and calculating the difference between the center point coefficient value and a preset maximum value to obtain a coefficient space; dividing the coefficient space in the sharpening filter matrix by controlling diagonal and quadruple neighbor points according to a preset ratio to obtain the edge coefficient configuration of the sharpening filter matrix; the center point coefficient value and the edge coefficient configuration of the sharpening filter matrix constitute the first proportional coefficient configuration for sharpening processing.
7. The image scaling optimization device based on a matrix system as described in claim 6, characterized in that, The integrated module includes an acquisition unit, a difference unit, an equalization unit, and a construction unit; The acquisition unit is used to acquire the center point coefficient value sent by the client. The difference unit is used to take the center point coefficient value as the center point coefficient of the smoothing filter matrix, and calculate the difference between the center point coefficient value and the preset maximum value of the coefficient to obtain the coefficient space. The equal division unit is used to control all neighborhood edge points to equally divide the coefficient space in the smoothing filter matrix according to a preset ratio, so as to obtain the edge coefficient configuration of the smoothing filter matrix. The construction unit is configured to form the second scaling factor configuration for smoothing processing by the center point coefficient values and the edge coefficient configuration of the smoothing filter matrix.
8. A storage medium, characterized in that, The storage medium stores a computer program, which is called and executed by a computer to implement a scaling image optimization method based on a matrix system as described in any one of claims 1 to 5.