Image processing device, method of operation of an image processing device, and storage medium

Abstract

The present application provides an image processing device, an image processing device working method and a storage medium. The image processing device comprises: a first judging unit (21) for judging the change level of an image through a first AI operation; a comparison unit (11) for comparing the change level with a specified value; a second judging unit (22) for judging which scene of multiple scenes the image is classified into through a second AI operation only when the change level exceeds the specified value; a setting unit (12) for setting a quality parameter based on the judged scene; and an adjusting unit (13) for adjusting the image using the quality parameter.

Description

[0001] This application claims priority to Japanese Patent Application No. 2021-006896, filed on January 20, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to image processing equipment, methods of operating the image processing equipment, and storage media. Background Technology

[0003] Develop a video display device such as a television receiver or smartphone that automatically adjusts the picture and sound quality of a program to match its genre. Program genres are obtained from raw data such as Electronic Program Guide (EPG) schedules.

[0004] However, a program may contain multiple scenes (scenes) with varying levels of visual quality. For example, a news program may include scenes featuring people, indoor scenes, scenic scenes, and sports scenes.

[0005] Therefore, adjustments based solely on program type cannot provide footage best suited to each scenario. Furthermore, there are also instances where original data such as the EPG program schedule cannot be referenced.

[0006] In recent years, artificial intelligence (AI) computing has been used in image processing. AI computing is computationally intensive, thus requiring significant resources (computing resources). Methods for efficiently performing AI computing are currently under development.

[0007] However, peripheral devices such as television receivers have limited resources, making it difficult to use AI computing for appropriate image processing.

[0008] Prior art literature

[0009] Patent documents

[0010] Patent Document 1: Japanese Patent Application Publication No. 2008-28871

[0011] Patent Document 2: International Publication No. 2018 / 067962 Summary of the Invention

[0012] The purpose of this application is to provide an image processing device that outputs the most appropriate image corresponding to a scene, a method for operating the image processing device that outputs the most appropriate image corresponding to a scene, and a non-volatile storage medium storing an image processing program that outputs the most appropriate image corresponding to a scene.

[0013] The image processing apparatus provided in this application includes: a first judgment unit that judges the change level of an image through a first AI calculation; a comparison unit that compares the change level with a predetermined value; a second judgment unit that, only when the change level exceeds the predetermined value, judges which scene among multiple scenes the image is classified into through a second AI calculation; a setting unit that sets image quality parameters based on the judged scene; and an adjustment unit that adjusts the image using the image quality parameters.

[0014] The image processing device provided in this application includes the following operating method: determining the change level of an image through a first AI calculation; comparing the change level with a predetermined value; determining, through a second AI calculation, which scene the image is classified into among multiple scenes only if the change level exceeds the predetermined value; setting image quality parameters based on the determined scene; and adjusting the image using the image quality parameters.

[0015] The present application provides a computer-readable non-volatile storage medium storing an image processing program that enables a computer to perform the following steps: determining the level of change in an image through a first AI operation; comparing the level of change with a predetermined value; determining, through a second AI operation, which scene the image is classified into among multiple scenes only if the level of change exceeds the predetermined value; setting image quality parameters based on the determined scene; and adjusting the image using the image quality parameters. Attached Figure Description

[0016] Figure 1 It is a structural diagram of a television receiver that includes the image processing apparatus of the embodiment;

[0017] Figure 2 This is a flowchart of the operation method of the image processing apparatus according to the first embodiment;

[0018] Figure 3 This is a diagram used to explain the operation method of the image processing apparatus according to the first embodiment;

[0019] Figure 4 This is a flowchart of the operation method of the image processing apparatus according to the second embodiment;

[0020] Figure 5 This is a diagram used to explain the operation method of the image processing apparatus according to the second embodiment. Detailed Implementation

[0021] First Implementation Method

[0022] like Figure 1As shown, in this embodiment, the image processing device 1, tuner 31, and memory 32 constitute a receiving device 30, and the receiving device 30, display 42, and speaker 43 constitute a receiving system 9. The receiving device 30 may also be a television receiving device integrated with the display 42 and speaker 43.

[0023] The display 42 can be a liquid crystal display (LCD), an electroluminescent (EL) display, a plasma display, a surface-conduction electro-emitter display (SED), a video projector, a rear projection display, or a cathode ray tube (including flat-panel types), etc. The terminal for the user to operate the receiving device 30, i.e., the remote control 44, can be a smartphone, a tablet terminal, an AI speaker, etc.

[0024] Tuner 31 receives signals, for example, by selecting one channel from multiple channels of terrestrial digital television broadcasts and satellite digital television broadcasts received by receiving antenna 41. Tuner 31 can also receive Internet broadcasts input from server 47 via network line 46. Program footage recorded in video recorder 45 can also be input to receiving device 30.

[0025] The image processing device 1 processes the input image and outputs image and sound signals. The image signal is output to the display 42, and the sound signal is output to the speaker 43, so that the user can watch the program.

[0026] The image processing device 1 includes a CPU 10 as a processor and an AI computing unit 20 as a neural network.

[0027] The AI ​​computing unit 20 has a first determination unit 21 and a second determination unit 22. The first determination unit 21 and the second determination unit 22 share the resources of the AI ​​computing unit 20, and therefore cannot perform computing processing simultaneously. The AI ​​computing unit 20 includes semiconductors, for example, to read programs stored in the memory 32 and perform operations.

[0028] As will be described later, the first determination unit 21 includes a first AI operation (AI operation 1) that uses a neural network to determine the image change level D of the image. The second determination unit 22 includes a second AI operation (AI operation 2) that uses a neural network to determine which of the multiple scenes the image is classified into.

[0029] Neural network-based AI operations utilize deep learning algorithms to perform image analysis. Deep learning algorithms include well-known methods such as Convolutional Neural Networks (CNNs), fully connected layers, and output layers. Deep learning is also known as deep learning technology. Since image analysis based on deep learning is a well-known technique, detailed explanations are omitted.

[0030] The CPU 10 performs overall control of the receiving device 30. The CPU 10 includes semiconductors, for example, to read programs stored in the memory 32 and perform operations. The CPU 10 includes a comparison unit 11, a setting unit 12, and an adjustment unit 13. Furthermore, at least any of these functional units executed by the CPU 10 can be configured as a dedicated circuit independent of the CPU 10. Alternatively, a single CPU unit may have both a CPU 10 and an AI processing unit 20. However, for high-speed processing, it is preferable that AI calculations are performed in a dedicated AI processor.

[0031] The comparison unit 11 compares the change level D of the image determined by the first judgment unit 21 with the predetermined value K. The second judgment unit 22 performs a scene judgment calculation, i.e., a second AI calculation, only if the change level D determined by the first judgment unit 21 through the first AI calculation exceeds the predetermined value K.

[0032] For example, if the specified value K is 75% and the probability of image change, i.e., the change level D, is 80%, then a second AI calculation is performed because the change level D exceeds the specified value K. The setting unit 12 sets image quality parameters based on the scene determined by the second judgment unit 22. The adjustment unit 13 uses the image quality parameters to adjust the image.

[0033] When the specified value K is 75% and the change level D is 60%, the second AI calculation is not performed because the change level D is below the specified value K.

[0034] In existing AI computing, multiple operations must be performed sequentially in a pipeline manner. That is, the second AI operation is performed regardless of the output of the first AI operation. In contrast, in image processing device 1, the second AI operation may be performed based on the output of the first AI operation, without performing the second AI operation. Therefore, even a peripheral device with limited resources, i.e., an image processing device, can output the most appropriate image corresponding to the scene.

[0035] like Figure 2 The working method of the image processing device 1 provided in this application will be described.

[0036] Step S10: Input frame image.

[0037] like Figure 3As shown in the previous paragraph, a television broadcast image may have 30 frames (still images) in 1 second. The first determination unit 21 is input with the frame image (first image) and its next frame image (second image).

[0038] Step S20: First AI operation (AI operation 1).

[0039] The first judgment unit 21 performs a first AI operation, in which the AI ​​calculation unit 20 determines the change level D between the first image and the second image. For example, in the first AI operation, a two-dimensional feature map is extracted or a one-dimensional feature vector is extracted.

[0040] If scene determination is based on changes in image brightness or the brightness of each pixel, there's a risk that slight zooming in or camera rotation might incorrectly identify scene changes. However, by using AI calculations, scene changes can be accurately determined.

[0041] Step S30: Compare the levels of change (determine if the level of change is greater than K).

[0042] The comparison unit 11 compares the change level D determined by the first judgment unit 21 with the predetermined value K. If the change level D is greater than the predetermined value K (yes), the process of step S40 is performed. If the change level D is less than or equal to the predetermined value K (no), the process of step S10 is performed.

[0043] Furthermore, if the specified value K is too small, there is a concern that frequent image quality adjustments will result in unnatural images. Therefore, the specified value K is set to an appropriate value, such as over 70%. The specified value K can also be changed through user operation.

[0044] Step S40: The second AI operation (AI operation 2) begins.

[0045] The second judgment unit 22 performs a second AI operation in the AI ​​calculation unit 20, which is shared with the first judgment unit 21, to determine which scene among multiple scenes the second image belongs to.

[0046] Scenes include, for example, scenes with people, landscapes, night scenes, and sports scenes.

[0047] For example, in the second AI operation, object detection or segmentation is performed with two-dimensional feature maps as input, or image classification is performed with one-dimensional feature vectors as input.

[0048] Step S50: Time measurement (after TA).

[0049] like Figure 3As shown, in the image processing apparatus 1, the processing interval (time) TA of the repeatedly performed first determination unit 21, i.e., the interval TA of the first AI operation, is longer than the first processing time T1 of the first AI operation. However, the interval TA is shorter than the total time of the first processing time T1 of the first AI operation and the second processing time T2 (T2A+T2B) of the second AI operation. Therefore, the second AI operation is not completed during the interval TA.

[0050] If the processing interval TA (yes) is reached, the image processing device 1 temporarily interrupts the second AI operation and performs processing starting from step S60.

[0051] Step S60: Input frame image.

[0052] Similar to step S10, two new frame images are input into the first determination unit 21.

[0053] Step S70: First AI operation (AI operation 1).

[0054] Similar to step S20, the first judgment unit 21 performs the first AI operation to determine the change level D.

[0055] Step S80: Compare the levels of change (determine if the level of change is greater than K).

[0056] Similar to step S30, the comparison unit 11 compares the change level D determined by the first determination unit 21 with the predetermined value K. In step S80, if the change level D is greater than the predetermined value K (yes), a new second AI operation is performed in step S40. The second operation that was processed midway is forcibly terminated. Alternatively, the intermediate result that has already been processed can be used as the result of the second AI operation. Conversely, if the change level D is less than or equal to the predetermined value K (no), the second AI operation A that was processed midway is restarted.

[0057] That is, the second AI operation based on the second determination unit 22 is performed in segments during the period when the first AI operation based on the first determination unit 21 is not performed. In the image processing device 1, the second AI operation is performed by dividing it into two parts, second AI operations 2A and 2B, but the second AI operation can also be divided into three or more parts.

[0058] Furthermore, in image processing device 1, the processing interval TA of the first AI operation is longer than the frame interval Tf (e.g., 1 / 30 of a second). However, if the AI ​​operation speed is fast, the first AI operation can also be performed on all frames of the image.

[0059] Furthermore, a third AI operation can be performed following the second AI operation. For example, after determining that the image scene is "moving" in the second AI operation, the specific match name "football" can be determined in the third AI operation.

[0060] Step S90: The second AI operation (AI operation 2) ends.

[0061] If the second AI operation ends (yes), then the series of processes starting from step S10 are performed again, while the process of step S100 is also performed. This continues until the second AI operation ends (no).

[0062] Step S100: Image quality settings / adjustments.

[0063] Based on the scene determined by the second determination unit 22, the setting unit 12 sets the image quality parameters. The adjustment unit 13 uses the image quality parameters to adjust the image, that is, the frame image after the frame image that has changed.

[0064] Image quality parameters include brightness, color saturation, shadows, color temperature, sharpness, noise reduction level, contrast enhancement level, and detail enhancement level.

[0065] For example, in landscape scenes, a vivid image is achieved by increasing the brightness, color saturation, and shadow levels above standard parameters. In portrait scenes, skin texture becomes more natural by increasing noise reduction and detail enhancement levels while decreasing color saturation levels. Image quality parameters for each scene are pre-stored in memory 32.

[0066] Image processing device 1 is a peripheral device with limited resources, but it can still output the most appropriate image for the scene.

[0067] As described above, the working method of the image processing device provided in this application includes: step S20 of determining the change level of an image through a first AI calculation; step S30 of comparing the change level with a predetermined value; step S40 of determining which scene among multiple scenes the image is classified into through a second AI calculation only when the change level exceeds the predetermined value; step S100 of setting image quality parameters based on the determined scene; and step S100 of adjusting the image using the image quality parameters.

[0068] The image processing program provided in this application enables a computer to perform the following steps: step S20, determining the level of change of an image through a first AI operation; step S30, comparing the level of change with a predetermined value; step S40, determining which scene among multiple scenes the image is classified into through a second AI operation only if the level of change exceeds the predetermined value; step S100, setting image quality parameters based on the determined scene; and step S100, adjusting the image using the image quality parameters.

[0069] Variation 1 of the first embodiment

[0070] The image processing device 1A in this modified example is similar to the image processing device 1, therefore the same symbols are assigned to structural elements with the same function and the description is omitted.

[0071] The image processing device 1A acquires, for example, program data (e.g., EPG) of the image signal attached to a television program. The EPG data is stored in memory 32. In addition to program name, performers, program summary, etc., the program data also includes category data. Categories include, for example, "News / Report," "Sports," "Information / Slideshow," "Drama," "Music," "Variety," "Movie," "Motion Picture / Special Effects," "Documentary / Culture," "Theater / Performance," "Entertainment / Education," and "Benefits."

[0072] The setting unit 12 of the image processing device 1A sets image quality parameters based on the type and the scene determined by the second determination unit 22.

[0073] In other words, even for the same scenic scene, news footage uses smaller increases in brightness, color saturation, and shadow levels compared to cinematic footage. Therefore, news footage doesn't show a significant difference when switching from a scene with people to a scenic one. Conversely, cinematic footage outputs more impactful and impactful scenic scenes than news footage.

[0074] Image quality parameters for various scenes of multiple types are pre-stored in memory 32. The image processing device 1A can adjust the image of the scene more appropriately according to the type.

[0075] Variation 2 of the first embodiment

[0076] Since the image processing device 1B in this modified example is similar to the image processing device 1, the same symbols are assigned to structural elements with the same function and the descriptions are omitted.

[0077] The setting unit 12 of the image processing device 1B sets not only image quality parameters but also sound quality parameters based on the scene. The adjustment unit 13 adjusts not only the image quality but also the sound of the image using the sound quality parameters.

[0078] Audio quality parameters include, for example, equalizer levels and noise reduction levels based on high-pass and low-pass filters.

[0079] For example, in a conversational scene where a character's mouth is moving, the equalizer level is set to flat and the noise reduction level is set to high to make it easier to hear.

[0080] In the image processing device 1B, the image is not only adjusted, but the sound is also adjusted appropriately according to the scene.

[0081] In the image processing device 1B, the image quality parameters can, of course, be set based on the type and the scene determined by the second determination unit 22, just like in the image processing device 1A.

[0082] Second Implementation Method

[0083] The image processing device 1C in this embodiment is similar to the image processing device 1, etc., so the same symbols are given to structural elements with the same function and the description is omitted.

[0084] like Figure 4 The working method of the image processing device 1C provided in this application will be described.

[0085] Steps S10-S30

[0086] and Figure 2 The image processing device 1 described herein is the same.

[0087] Step S41

[0088] The second AI operation begins and is processed until it ends. After the second AI operation ends, the series of processes that started from step S10 are performed again, while the processing of step S100 is also performed.

[0089] Step S100

[0090] and Figure 2 The image processing device 1 described herein is the same.

[0091] In the image processing apparatus 1C, similar to the image processing apparatus 1, the first processing interval (time) TA of the first judgment unit 21, which is repeatedly performed, is shorter than the total time of the first processing time T1 of the first AI operation and the second processing time T2 (T2A+T2B) of the second AI operation.

[0092] like Figure 5As shown, the first determination unit 21 of the image processing apparatus 1C does not restart processing until the processing of the second determination unit 22 is completed. Therefore, the second processing interval TA2 of the first determination unit 21 when the second AI operation is performed is longer than the first processing interval TA1 when the second AI operation is not performed.

[0093] Because the processing interval of the first determination unit 21 in the image processing device 1C may become longer, there is concern that the image may not be able to adjust the scene appropriately in images with drastic scene changes. However, when the first determination unit 21 detects a scene change in the image, it can output an image with more appropriate image quality faster than the image processing device 1.

[0094] In the image processing device 1C, image quality adjustment can be performed based on the type and the scene determined by the second determination unit 22, as in the image processing device 1A, or sound quality adjustment can be performed based on the scene, as in the image processing device 1B.

[0095] This application also provides a computer-readable non-volatile storage medium storing computer instructions that, when executed by a processor, implement the image quality processing described above.

[0096] Several embodiments of this application have been described, but these embodiments are provided as examples and are not intended to limit the scope of this application. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of this application. These embodiments and their variations are included within the scope and spirit of this application, and are included within the scope of this application as set forth in the claims and its equivalents.

Claims

1. An image processing device, characterized in that, include: The first judgment unit extracts features of the image through a first AI operation and judges the change level of the image based on the features. The features include a two-dimensional feature map or a one-dimensional feature vector. The image information represented by the two-dimensional feature map or the one-dimensional feature vector does not include the brightness information of the image. The comparison section compares the change level with a specified value; The second judgment unit determines which scene among multiple scenes the image is classified into only when the change level exceeds the predetermined value, through a second AI operation. If the feature is a two-dimensional feature map, the second AI operation is used to perform object detection or segmentation on the two-dimensional feature map. If the feature is a one-dimensional feature vector, the second AI operation is used to perform image classification processing on the one-dimensional feature vector. The settings department sets image quality parameters based on the identified scene. and The adjustment unit adjusts the image using the aforementioned image quality parameters.

2. The image processing apparatus according to claim 1, characterized in that, in, The processing interval of the first judgment unit is longer than the first processing time of the first judgment unit, and the processing interval of the first judgment unit is shorter than the combined time of the first processing time and the second processing time of the second judgment unit. The second determination unit, which shares resources with the first determination unit, performs processing during periods when the first determination unit is not processing.

3. The image processing apparatus according to claim 1, characterized in that, in, The first processing interval of the first judgment unit is longer than the first processing time of the first judgment unit, and the first processing interval of the first judgment unit is shorter than the combined time of the first processing time and the second processing time of the second judgment unit. The first decision unit does not restart processing until the processing of the second decision unit for the shared resource is completed.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that, in, The image processing equipment, together with the tuner, display, and speaker, constitutes a television receiving device.

5. The image processing apparatus according to any one of claims 1 to 4, characterized in that, in, The images are images of broadcast programs of a specific type; The setting unit sets the image quality parameters based on the type and the scene determined by the second determination unit.

6. The image processing apparatus according to any one of claims 1 to 5, characterized in that, in, The setting unit sets the sound quality parameters based on the scene. The adjustment unit uses the sound quality parameters to adjust the sound of the image.

7. A method of operating an image processing device, characterized in that, include: The image features are extracted through the first AI operation, and the change level of the image is determined based on the features. The features include a two-dimensional feature map or a one-dimensional feature vector. The image information represented by the two-dimensional feature map or the one-dimensional feature vector does not include the brightness information of the image. Compare the change level with the specified value; Only when the change level exceeds the specified value, the second AI operation determines which scene the image is classified into among multiple scenes. If the feature is a two-dimensional feature map, the second AI operation is used to perform object detection or segmentation on the two-dimensional feature map. If the feature is a one-dimensional feature vector, the second AI operation is used to perform image classification processing on the one-dimensional feature vector. Based on the identified scene, set the image quality parameters; and Adjust the image using the aforementioned image quality parameters.

8. A computer-readable non-volatile storage medium, characterized in that, The storage medium stores a program or computer instructions that are executed on a computer, wherein the computer performs the following steps: The image features are extracted through the first AI operation, and the change level of the image is determined based on the features. The features include a two-dimensional feature map or a one-dimensional feature vector. The image information represented by the two-dimensional feature map or the one-dimensional feature vector does not include the brightness information of the image. Compare the change level with the specified value; Only when the change level exceeds the specified value, the second AI operation determines which scene the image is classified into among multiple scenes. If the feature is a two-dimensional feature map, the second AI operation is used to perform object detection or segmentation on the two-dimensional feature map. If the feature is a one-dimensional feature vector, the second AI operation is used to perform image classification processing on the one-dimensional feature vector. Based on the identified scene, set the image quality parameters; and Adjust the image using the aforementioned image quality parameters.

Images