Image processing device and image processing method

Abstract

This image processing device comprises an image conversion unit that performs aspect ratio conversion by image expansion on an image to be recognized by an image recognition AI model, which is an AI model for performing image recognition processing, before the image is resized to an input tensor size of the image recognition AI model.

Description

Image processing apparatus, image processing method

[0001] The present technology relates to an image processing apparatus and method, and particularly to an image processing technology for an image input to an AI (Artificial Intelligence) model.

[0002] An image recognition system that performs AI (Artificial Intelligence) processing related to recognition of image content such as object detection processing and object recognition processing on a captured image has been put into practical use. In such an image recognition system, resizing the captured image to the input tensor size of the AI model is performed. For example, resizing a captured image with a frame size of 1920×1080 to a frame size of 320×320.

[0003] Regarding related prior art, Patent Document 1 below can be cited. Patent Document 1 below discloses a technique for changing the aspect ratio of default boxes and label data in the generation of teacher data in machine learning of an object detection model such as SSD (Single Shot multibox Detector).

[0004] International Publication No. 2023 / 085190

[0005] Here, the resizing described above is also performed on training images during the training of the AI ​​model. However, the image size of the training images and the image size of the captured images used during inference do not necessarily match. For example, the image size of the training images may be 4000 x 3000, while the image size of the captured images may be 1920 x 1080. When the image sizes of the training images and captured images do not match and their aspect ratios differ, there is a risk that the inference performance will decrease. Specifically, if the aspect ratios of the training images and captured images differ in this way, the aspect ratio of the subject in the resized image will differ between training and inference. As a result, even though training was performed on a subject with an aspect ratio of a:b, inference will be performed on a subject with an aspect ratio of c:d, which may lead to a decrease in inference performance.

[0006] This technology was developed in light of the above-mentioned challenges, and aims to improve the inference performance of AI models that perform image recognition processing.

[0007] The image processing device related to this technology includes an image transformation unit that performs aspect ratio transformation by image augmentation on an image that is the target of recognition by an image recognition AI model, which is an AI model that performs image recognition processing, before it is resized to the input tensor size of the image recognition AI model. This technology is applicable to both inference and training. If aspect ratio transformation by image augmentation is performed as described above during inference, it becomes possible to adjust the aspect ratio of the target image to the aspect ratio of the training input image of the AI ​​model before resizing, thereby preventing the aspect ratio of the subject in the image used as the input tensor of the AI ​​model from differing between training and inference, and improving inference performance. Furthermore, if aspect ratio transformation by image augmentation is performed as described above during training, it becomes possible to provide the image recognition AI model's learner with multiple images with different aspect ratios as training input images, making it possible to generate an AI model that can absorb differences in the aspect ratio of subjects, and improving inference performance.

[0008] This is an overview diagram of an inference system configured with an image processing device as an embodiment. This is a block diagram showing an example configuration of an image processing device (imaging device) as a first embodiment. This is a block diagram showing an example hardware configuration of a server device included in an inference system as an embodiment. This is an explanatory diagram about resizing during training. This is an explanatory diagram about resizing during inference. This is an explanatory diagram about the operation of the image conversion unit. This is a diagram showing an example of the internal configuration of an image conversion unit that realizes image augmentation using an AI model. This is an explanatory diagram of a specific example of image augmentation. This is a flowchart showing a specific example of a processing procedure for realizing the image conversion method as a first embodiment. This is a block diagram showing an example configuration of an image processing device (learning device) as a second embodiment. This is a diagram showing an example configuration of an image conversion unit as a modified example.

[0009] The embodiments of this technology will be described below in the following order, with reference to the attached drawings. <1. First Embodiment> (1-1. Overview of the inference system as an embodiment) (1-2. Example of the configuration of the imaging device) (1-3. Example of the configuration of the server device) (1-4. Image conversion method as the first embodiment) (1-5. Processing procedure) <2. Second Embodiment> <3. Modified Examples> <4. Summary of Embodiments> <5. This Technology>

[0010] <1. First Embodiment> (1-1. Overview of the Inference System as an Embodiment) Figure 1 is an overview diagram of an inference system configured with an image processing device as an embodiment of the present technology. As shown in the figure, the inference system of the embodiment comprises an imaging device 1, a server device 2, and a user terminal 3. In this inference system, the imaging device 1 corresponds to one embodiment of the image processing device according to the present technology.

[0011] In the inference system, the server device 2 and the user terminal 3 are configured as computer devices equipped with microcomputers having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), respectively. In this example, the server device 2 is configured to be able to communicate data with the imaging device 1 and the user terminal 3 via a network NT, such as the Internet.

[0012] The imaging device 1 captures an image of a subject and obtains an image. Here, "imaging" in this specification broadly means obtaining image data that captures a subject. The image data referred to here is a general term for data consisting of multiple pixel data, and the pixel data is a broad concept that includes not only data indicating the amount of light received from the subject, but also data indicating, for example, the distance to the subject, the polarization information of the subject, and temperature information. In other words, the "image data" obtained by "imaging" (image data) includes data as a grayscale image that shows the amount of light received for each pixel, data as a distance image that shows the distance to the subject for each pixel, data as a polarization image that shows the polarization information of incident light for each pixel, and data as a thermal image that shows temperature information for each pixel. Furthermore, the "image data" also includes data as an event image obtained by an EVS (Event-based Vision Sensor) having an event sensor in which multiple event detection pixels that detect changes in the amount of light received as events are arranged in two dimensions. This event image data can be rephrased as image data that shows whether or not an event has occurred for each pixel, and can be expressed as image data that captures the movement of the subject.

[0013] As an example, the imaging device 1 in this example is configured to obtain the above-mentioned grayscale image as the captured image, similar to a typical digital camera. As shown in the figure, the inference system in this example is provided with multiple imaging devices 1.

[0014] In the inference system, server device 2 is a computer device intended to be used by the provider of a service using the inference system. User terminal 3 is a computer device intended to be used by the user who receives the service.

[0015] The inference system in this example is configured to provide a service that performs inference processing using an AI (Artificial Intelligence) model on captured images obtained by the imaging device 1, generates analytical information showing the analysis results of the subject based on the inference results, and presents it to the user.

[0016] Here, assuming a service that presents the user with analysis information of the subject based on the inference results as described above, various applications for the imaging device 1 can be considered as surveillance cameras. For example, applications include surveillance cameras for indoors such as stores, offices, and residences; surveillance cameras for monitoring outdoors such as parking lots and streets (including traffic surveillance cameras, etc.); surveillance cameras for manufacturing lines in FA (Factory Automation) and IA (Industrial Automation); and surveillance cameras for monitoring inside and outside vehicles.

[0017] For example, in the case of surveillance cameras in a store, multiple imaging devices 1 could be placed at designated locations within the store, allowing users to observe the demographics of customers (such as gender and age group) and their behavior within the store (movement patterns). In this case, the analysis information could include data on customer demographics, movement patterns within the store, and congestion at checkout counters (e.g., waiting times at checkout counters). Alternatively, in the case of traffic surveillance cameras, each imaging device 1 could be placed at various locations near the road, allowing users to recognize information such as license plates, vehicle colors, and vehicle types of passing vehicles. In this case, the analysis information could include data on license plates, vehicle colors, and vehicle types.

[0018] Furthermore, if traffic surveillance cameras are used in a parking lot, the cameras should be positioned to monitor each parked vehicle, checking for suspicious individuals behaving inappropriately around each vehicle. If a suspicious person is detected, the system could notify users of their presence and their characteristics (gender, age group, clothing, etc.). Additionally, it is conceivable to monitor vacant spaces in the city and parking lots and notify users of available parking spaces.

[0019] In this example, the inference processing targeting the captured image is performed by the imaging device 1. Specifically, the imaging device 1 is equipped with an AI model for performing inference processing, and information indicating the results of the inference processing using the AI ​​model (hereinafter referred to as "inference result information") is transmitted to the server device 2. Based on the inference result information transmitted from the imaging device 1 in this manner, the server device 2 performs the various normalization processes described above.

[0020] By employing this method in which the imaging device 1 performs inference processing and transmits the inference result information to the server device 2, the amount of communication data required to perform inference processing can be significantly reduced compared to the case in which the server device 2 performs inference processing on captured images transmitted from the imaging device 1. Furthermore, since it is no longer necessary to transmit captured images from the imaging device 1 to the server device 2, it is possible to prevent the leakage of captured images containing personal information to external parties, thereby protecting privacy.

[0021] In this example, the AI ​​model used by the imaging device 1 is transmitted from the server device 2 to the imaging device 1. In this example, the imaging device 1 is owned by the user, and the user can transmit the AI ​​model from the server device 2 to the imaging device 1 by purchasing the right to use the AI ​​model through payment of a fee to the service provider.

[0022] It should be noted that this premise is merely an example for illustrative purposes, and the AI ​​model may be one that is pre-configured in the imaging device 1.

[0023] Although Figure 1 shows an example where the inference system is equipped with multiple imaging devices 1, the number of imaging devices 1 only needs to be at least one. Also, although Figure 1 shows that there is one user terminal 3 in the inference system, there may be multiple user terminals 3. In other words, it is conceivable that there may be multiple users receiving services from the inference system.

[0024] (1-2. Example of Imaging Device Configuration) Figure 2 is a block diagram showing an example of the configuration of the imaging device 1. As shown in the figure, the imaging device 1 includes an image sensor 10, an imaging optical system 11, an optical system drive unit 12, a camera control unit 13, a memory unit 14, and a communication unit 15. The image sensor 10, the camera control unit 13, the memory unit 14, and the communication unit 15 are connected via a bus 16 and are capable of communicating data with each other.

[0025] In this example, the image sensor 10 is configured as a grayscale image sensor that obtains the aforementioned grayscale image. Specifically, the image sensor 10 is configured as a solid-state image sensor such as a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type.

[0026] The imaging optical system 11 includes lenses such as a cover lens, zoom lens, and focus lens, as well as an aperture (iris) mechanism. This imaging optical system 11 guides light (incident light) from the subject and focuses it onto the light-receiving surface (imaging surface) of the image sensor 10.

[0027] The optical system drive unit 12 comprehensively represents the drive units for the zoom lens, focus lens, and aperture mechanism of the imaging optical system 11. Specifically, the optical system drive unit 12 includes actuators for driving the zoom lens, focus lens, and aperture mechanism, and drive circuits for said actuators.

[0028] The camera control unit 13 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the imaging device 1 by having the CPU execute various processes according to a program stored in the ROM or a program loaded into the RAM.

[0029] Furthermore, the camera control unit 13 issues drive instructions to the optical system drive unit 12 for the zoom lens, focus lens, aperture mechanism, etc. In response to these drive instructions, the optical system drive unit 12 will perform actions such as moving the focus lens and zoom lens, and opening and closing the aperture blades of the aperture mechanism.

[0030] Furthermore, the camera control unit 13 controls the writing and reading of various data to and from the memory unit 14. The memory unit 14 is a non-volatile storage device such as an HDD (Hard Disk Drive) or flash memory device, and is used to store data used by the camera control unit 13 when performing various processes. The memory unit 14 can also be used as a storage location (recording location) for image data output from the image sensor 10.

[0031] The camera control unit 13 performs various data communications with external devices via the communication unit 15. In this example, the communication unit 15 is configured to enable communication via the network NT shown in Figure 1, and to perform data communications with external devices connected to the network NT, particularly in this example, at least with the server device 2.

[0032] As shown in the figure, the image sensor 10 includes an imaging unit 41, an image signal processing unit 42, an internal sensor control unit 43, an AI processing unit 44, a memory unit 45, a communication interface (I / F) 46, a pre-processing unit 47, a post-processing unit 48, and an image conversion unit 49, each of which is connected via a bus 50 and is capable of data communication with one another.

[0033] The imaging unit 41 includes a pixel array unit in which pixels having photoelectric conversion elements (light-receiving elements) such as photodiodes are arranged in two dimensions, and a readout circuit that reads out electrical signals (received signals) obtained by photoelectric conversion from each pixel of the pixel array unit. In this readout circuit, the electrical signals obtained by photoelectric conversion are subjected to processes such as CDS (Correlated Double Sampling) processing and AGC (Automatic Gain Control) processing, and further A / D (Analog to Digital) conversion processing.

[0034] The image signal processing unit 42 performs preprocessing, syncing, YC generation, and codec processing on the captured image signal as digital data after A / D conversion. Preprocessing includes clamping, which clamps the black levels of R (red), G (green), and B (blue) to predetermined levels, and correction processing between R, G, and B color channels. Preprocessing can also include brightness adjustment processing such as gamma correction and color adjustment processing such as white balance adjustment. In syncing processing, color separation processing is performed so that the image data for each pixel has all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaicing is performed as the color separation processing. In YC generation processing, luminance (Y) signals and color (C) signals are generated (separated) from the R, G, and B image data. In codec processing, the image data that has undergone the above various processing is subjected to encoding processing for recording or communication, and file generation, for example. Codec processing allows for the creation of video files in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264. It also allows for the creation of still image files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).

[0035] The sensor control unit 43 is configured with a microcomputer, for example, which includes a CPU, ROM, RAM, etc., and comprehensively controls the operation of the image sensor 10. For example, the sensor control unit 43 issues instructions to the imaging unit 41 to control the execution of imaging operations. It also controls the execution of processing to the image signal processing unit 42.

[0036] The AI ​​processing unit 44 is configured with a programmable computing device such as a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array), and performs inference processing (AI processing) on ​​the captured image using an AI model.

[0037] In this example, the AI ​​model used by the AI ​​processing unit 44 is an image recognition AI model that performs image recognition processing. Image recognition processing here refers to the process of recognizing the content of an image. Examples of image recognition processing include object detection processing that detects the region in which an object exists, object recognition processing that recognizes what kind of object is depicted in the image, semantic segmentation processing, and anomaly detection processing such as PatchCore. In addition, for object detection processing, such as YOLO (You Only Look Once) and SSD (Single Shot Multibox Detector), processing may be performed not only to detect the region in which an object exists, but also to recognize what kind of object it is.

[0038] The memory unit 45 is used to hold (temporarily store) data necessary for AI processing by the AI ​​processing unit 44. Specifically, the memory unit 45 is used to store AI model data, which is the necessary parameter data for constructing an AI model (image recognition AI model) in the AI ​​processing unit 44. If the AI ​​model has a neural network such as a CNN (Convolutional Neural Network), this AI model data corresponds to parameters that indicate the structure of the neural network, or parameters that serve as filter coefficients used in convolution processing, etc. In this example, the memory unit 45 is also used to hold captured images that become input data to the AI ​​processing unit 44.

[0039] Furthermore, the memory unit 45 in this example is also used to store the learning image information IL, which will be explained in more detail later.

[0040] The communication interface 46 is an interface that communicates with various parts connected via the bus 16, such as the camera control unit 13 and the memory unit 14, located outside the image sensor 10. For example, the communication interface 46 communicates to acquire AI model data, etc., from an external source, based on the control of the sensor's internal control unit 43, in order to realize the AI ​​model in the AI ​​processing unit 44. Furthermore, it is possible to output AI processing result information (inference result information), etc., from the AI ​​processing unit 44 to the outside of the image sensor 10 via the communication interface 46.

[0041] The preprocessing unit 47 performs predetermined image signal processing as preprocessing on the image input to the image recognition AI model. This preprocessing includes, for example, normalization of brightness values ​​and resizing to the size of the input tensor of the image recognition AI model, as well as image format conversion processing to fit the input tensor of the image recognition AI model.

[0042] The post-processing unit 48 performs predetermined signal processing as post-processing on the output of the image recognition AI model. The content of this post-processing may vary depending on the content of the inference task of the image recognition AI model. For example, when the image recognition AI model performs object detection processing, the post-processing unit 48 performs processing such as drawing the information of the bounding box (and the class name of the object) output by the image recognition AI model on the original image (the image to be inferred). Also, when the image recognition AI model performs semantic segmentation processing, the post-processing unit 48 performs processing such as drawing segment information based on the class identification value for each pixel output by the image recognition AI model on the original image. Thus, as post-processing, it includes at least the processing of drawing the output of the AI model on the original image.

[0043] The image conversion unit 49 performs predetermined image conversion processing on the image input to the preprocessing unit 47. Note that the image conversion unit 49 will be described later in detail.

[0044] (1-3. Configuration Example of Server Device) FIG. 3 is a block diagram showing a hardware configuration example of the server device 2. It is conceivable that the computer device as the user terminal 3 shown in FIG. 1 also adopts a hardware configuration similar to that shown in FIG. 3.

[0045] As shown in the figure, the server device 2 includes a CPU 21. The CPU 21 executes various processes according to a program stored in the ROM 22 or a program loaded from the storage unit 29 to the RAM 23. The RAM 23 also appropriately stores data and the like necessary for the CPU 21 to execute various processes.

[0046] The CPU 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input / output interface (I / F) 25 is also connected to this bus 24.

[0047] An input / output interface 25 is connected to an input unit 26 composed of an operator or an operation device. For example, as the input unit 26, various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. are assumed. An operation of a user is detected by the input unit 26, and a signal corresponding to the input operation is interpreted by a CPU 21.

[0048] Also, to the input / output interface 25, a display unit 27 composed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, etc., and an audio output unit 28 composed of a speaker, etc. are connected integrally or separately. The display unit 27 is used for displaying various information, and is constituted by, for example, a display device provided on a housing of a computer device, a separate display device connected to the computer device, etc.

[0049] The display unit 27 executes display of an image for various image processes, a moving image of a processing target, etc. on a display screen based on an instruction of the CPU 21. Also, the display unit 27 performs display as a GUI (Graphical User Interface), such as various operation menus, icons, messages, etc., based on an instruction of the CPU 21.

[0050] The input / output interface 25 may also be connected to a storage unit 29 composed of an HDD, a solid-state memory, etc., and a communication unit 30 composed of a modem, etc.

[0051] The communication unit 30 performs communication processing via a transmission path such as the Internet, communication with various devices by wired / wireless communication, bus communication, etc.

[0052] The input / output interface 25 is also connected with a drive 31 as required, and a removable recording medium 32 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.

[0053] The drive 31 can read data files such as programs used for various processes from the removable recording medium 32. The read data files are stored in the storage unit 29, or images and sounds contained in the data files are output by the display unit 27 and the sound output unit 28. Computer programs and other data read from the removable recording medium 32 are installed in the storage unit 29 as needed.

[0054] In a computer device having the hardware configuration described above, for example, the software for processing in this embodiment can be installed via network communication by the communication unit 30 or via the removable recording medium 32. Alternatively, the software may be stored in advance in the ROM 22 or storage unit 29, etc. The CPU 21 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing as a server device 2.

[0055] Furthermore, server device 2 is not limited to being a single computer device as shown in Figure 3, but may be configured as a system of multiple computer devices. These multiple computer devices may be systematized via a LAN (Local Area Network), or they may be located remotely via a VPN (Virtual Private Network) using the Internet, etc. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.

[0056] (1-4. Image conversion method as the first embodiment) In this image recognition system using an image recognition AI model, the image to be inferred is resized to the input tensor size of the AI ​​model (see the preprocessing unit 47 described above). Such resizing of the target image is performed not only during inference but also during training.

[0057] Figure 4 is an explanatory diagram regarding resizing during training. Here, an example is shown where an image with a size of 4000 x 3000 pixels, i.e., an aspect ratio of 4:3, is used as the training input image for an image recognition AI model. Through preprocessing, this training input image is resized to the input tensor size of the image recognition AI model, and the resized image is provided as input data for the learner. Here, the resized image size is assumed to be 320 x 320 pixels, and the aspect ratio is changed from 4:3 to 1:1.

[0058] Figure 5 is an explanatory diagram of resizing during inference using a trained image recognition AI model. During inference, the image to be inferred does not necessarily match the image size of the training input image, and the aspect ratio of the image may differ. In the figure, an example is shown where an image with a size of 1920 x 1080 pixels, i.e., an image with an aspect ratio of 16:9, is input as the image to be inferred. This image to be inferred is resized through preprocessing, that is, resized to a size of 320 x 320 pixels in this example, and the resized image is input to the inferencer (i.e., the trained image recognition AI model).

[0059] Here, Figures 4 and 5 illustrate the appearance of resized images, but as mentioned above, if the aspect ratio differs between the training input image and the image to be inferred, the aspect ratio of the target subject (the subject targeted for image recognition) in the resized image will differ between training and inference. Specifically, in the examples of Figures 4 and 5, the target subject becomes taller during inference than during training.

[0060] When an image with a different aspect ratio than the training input image is used as the image to be inferred, the training is performed on an object with an aspect ratio of a:b, but the inference is performed on an object with an aspect ratio of c:d, which may lead to a decrease in inference performance.

[0061] Therefore, in this embodiment, when the aspect ratio of the image to be inferred differs from the aspect ratio of the training input image, a method is adopted in which the aspect ratio of the image to be inferred is converted.

[0062] For this purpose, the imaging device 1 of this embodiment is provided with an image conversion unit 49 as shown in Figure 2. The image conversion unit 49 performs aspect ratio conversion by image augmentation on an image that is the target of recognition by the image recognition AI model and has not been resized to the input tensor size of the image recognition AI model. Specifically, the image conversion unit 49 in this example performs aspect ratio conversion by image augmentation on an RGB image obtained by demosaicing processing of the image signal processing unit 42.

[0063] In this specification, "image expansion" means increasing the image size by adding an image in at least one of the following directions: top, bottom, left, or right.

[0064] The image conversion unit 49 performs aspect ratio conversion, using the aspect ratio of the training input images used to train the image recognition AI model as the target aspect ratio. In other words, it converts the images that the image recognition AI model is intended to recognize to the same aspect ratio as the training input images.

[0065] Figure 6 is an explanatory diagram of the operation of the image conversion unit 49. Here, as in the examples of Figures 4 and 5 above, we illustrate the case where the aspect ratio of the training input image is 4:3 and the aspect ratio of the inference target image is 16:9. The image conversion unit 49 converts the 16:9 inference target image to the same 4:3 aspect ratio as the training input image, and the inference target image converted to a 4:3 aspect ratio in this way is input to the preprocessing unit 47. As a result, in the resizing process of the preprocessing unit 47 in this case, the reduction ratio of the image in the vertical and horizontal directions due to resizing is the same as during training, and it is possible to prevent the aspect ratio of the subject in the resized image from being different from that during training. Therefore, it is possible to prevent errors in the inference results caused by the aspect ratio of the subject being different from that during training, and to improve the inference performance of the image recognition AI model.

[0066] In this embodiment, information indicating the aspect ratio of the training input image is transmitted from the server device 2 and stored in the imaging device 1. Specifically, in this example, information indicating the aspect ratio of the training input image is stored in the memory unit 45 as training image information IL shown in Figure 2. The training image information IL only needs to be information indicating the aspect ratio of the training input image, and the specific information content is not limited. For example, the training image information IL could be information indicating the aspect ratio itself. Alternatively, it could be information on the image size (number of pixels in the vertical direction and number of pixels in the horizontal direction). Furthermore, it could be a combination of information indicating the aspect ratio itself and information on the image size. In this example, as an example, the training image information IL is assumed to be a combination of information indicating the aspect ratio itself and information on the image size.

[0067] In this example, the trained image recognition AI model is transmitted from the server device 2 and stored in the imaging device 1, but the training image information IL is transmitted from the server device 2 to the imaging device 1 together with the image recognition AI model data (the AI ​​model data mentioned above). This is merely an example, and the training image information IL may be transmitted from the server device 2 to the imaging device 1 at a different time than the AI ​​model data of the image recognition AI model, or it may be transmitted from a device other than the server device 2 to the imaging device 1. It is also conceivable that the training image information IL is stored in the imaging device 1 in advance. Furthermore, the training image information IL may be stored in a storage device other than the memory unit 45 (for example, the memory unit 14).

[0068] In this embodiment, the sensor control unit 43, in response to the state in which inference should be performed, determines whether the aspect ratio of the image to be inferred (in this example, the RGB image obtained by demosaicing of the image signal processing unit 42: the image input to the preprocessing unit 47) matches the aspect ratio of the input image for training, based on the learning image information IL. If the two do not match, it transmits the learning image information IL to the image conversion unit 49 and instructs it to perform aspect ratio conversion. If the two do match, the sensor control unit 43 inputs the image to be inferred to the preprocessing unit 47 without having the image conversion unit 49 process it.

[0069] The image conversion unit 49 performs aspect ratio conversion of the input image, that is, conversion to the aspect ratio of the training input image indicated by the training image information IL, in response to the above instruction from the sensor internal control unit 43.

[0070] This aspect ratio conversion is performed by image augmentation, and in this example, the image conversion unit 49 performs this image augmentation using an AI model. Specifically, the image conversion unit 49 performs image augmentation using an image inference AI model Me that infers images outside the frame based on the image content within the frame. By using the image inference AI model Me, which infers images outside the frame in this way, even if a part of the target subject extends beyond the frame in the original image, the image inference AI model Me can infer that part to be within the frame and obtain an augmented image (aspect ratio converted image). If an image in which a part of the target subject extends beyond the frame is input to the image recognition AI model of the AI ​​processing unit 44, there is a risk that the target subject may not be correctly recognized as the target subject. However, by inputting an augmented image that includes that part, the possibility of recognizing the target subject as the target subject can be increased, thereby improving the accuracy of the inference.

[0071] Figure 7 shows an example of the internal configuration of the image conversion unit 49 that realizes image augmentation using the AI ​​model as described above. As shown in the figure, the image conversion unit 49 is configured to have an AI processing unit 55 having an image inference AI model Me and a conversion control unit 56. The AI ​​processing unit 55 is configured to have a programmable arithmetic processing unit such as a DSP or FPGA, similar to the AI ​​processing unit 44 described above.

[0072] In this embodiment, when inferring an image outside the frame, it is desirable to prevent the image characteristics from differing between the inferred image portion (i.e., the extended portion) and the input image (captured image). If the image characteristics differ between the extended portion and the input image, there is a risk that the inference performance in the image recognition processing performed on the extended image will be reduced due to the difference in image characteristics.

[0073] Therefore, the image conversion unit 49 in this embodiment is configured to obtain an extended image in which noise based on sensor noise information indicating the nature of the sensor noise of the image sensor 10 is added to the extended image. Furthermore, the image conversion unit 49 in this embodiment is configured to obtain an extended image in which image distortion based on distortion information indicating the nature of the optical distortion occurring in the imaging optical system 11 is added to the extended image. Here, the sensor noise of the image sensor 10 in this example includes noise generated in the captured image due to noise superimposed on the light-receiving signal in the analog stage of the imaging unit 41.

[0074] In the image conversion unit 49 of this example, the functions for adding sensor noise and adding image distortion are provided to the image inference AI model Me.

[0075] Here, the function of the image inference AI model Me to infer an image outside the frame from the image content within the frame can be realized by performing machine learning in which the correct data of the extended portion of the training input image is provided as training data. In this case, in order to enable extension in all directions (up, down, left, and right), the training data provided is the correct data for each direction (up, down, left, and right). Furthermore, when providing the image inference AI model Me with the above-mentioned sensor noise addition function and image distortion addition function, in the machine learning in which the correct data of the extended portion is provided as training data, the sensor noise level and image distortion level are added as input to the learner, and training is performed using training data for each combination of assumed sensor noise level and image distortion level as training data. In this way, an image inference AI model Me can be realized that takes the captured image (RGB image obtained by demosaicing in this example by the image signal processing unit 42), the sensor noise level, and the image distortion level as inputs, and outputs an extended image in which noise and distortion corresponding to the sensor noise level and image distortion level specified for the extended portion are added.

[0076] In this example, the image inference AI model Me is configured to allow adjustment of the image augmentation width (augmentation rate) in each direction: up, down, left, and right. Specifically, it accepts input information indicating the augmentation width in each direction and adjusts the augmentation width in each direction according to that information.

[0077] In Figure 7, the conversion control unit 56 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs various controls related to image augmentation. Specifically, it determines whether the aspect ratio of the captured image and the training input image match based on the training image information IL, and provides instructions for augmentation to the AI ​​processing unit 55 (image inference AI model Me), as well as instructions for the sensor noise level and image distortion level.

[0078] Specifically, the conversion control unit 56 receives an instruction from the sensor-internal control unit 43 to perform aspect ratio conversion and sets the aspect ratio of the training input image identified from the training image information IL transmitted from the sensor-internal control unit 43 as the target aspect ratio. Then, it controls the AI ​​processing unit 55 so that image augmentation is performed based on this target aspect ratio.

[0079] In this example, image augmentation to match the target aspect ratio is achieved by first augmenting the image in each direction (up, down, left, and right) at a predetermined augmentation rate, and then, if necessary, augmenting in directions where augmentation is insufficient.

[0080] A specific example will be explained with reference to Figure 8. First, the conversion control unit 56 instructs the AI ​​processing unit 55 to perform image expansion in the up, down, left, and right directions of the captured image ("Im" in the figure) by a predetermined expansion rate (for example, 5%) (see "A1" in Figure 8A). At this time, the conversion control unit 56 instructs the AI ​​processing unit 55 to provide information on the sensor noise level and image distortion level of the expanded portion. Here, the sensor noise level and image distortion level information is stored as noise and distortion information in a predetermined storage device (for example, memory unit 14 or memory unit 45) within the imaging device 1, and the conversion control unit 56 obtains the sensor noise level and image distortion level information to be instructed to the AI ​​processing unit 55 from the storage device.

[0081] Here, the aspect ratio of the expanded image obtained by expanding in all directions (up, down, left, and right) as described above is referred to as the "current aspect ratio." The conversion control unit 56 determines whether the current aspect ratio matches the target aspect ratio, and if they do not match, it further expands the image in the directions necessary to match the target aspect ratio.

[0082] Specifically, if the current aspect ratio does not match the target aspect ratio, the conversion control unit 56 determines whether the current aspect ratio is larger than the target aspect ratio (i.e., whether it is wider than the target aspect ratio). If the current aspect ratio is larger than the target aspect ratio, it instructs the AI ​​processing unit 55 to perform vertical image expansion to match the target aspect ratio (see "A2" in Figure 8B). In this example, the vertical image expansion is performed equally in both the up and down directions, as shown in the figure.

[0083] On the other hand, if the current aspect ratio is not greater than the target aspect ratio (i.e., if it is more vertically elongated), the conversion control unit 56 instructs the AI ​​processing unit 55 to perform horizontal image expansion to match the target aspect ratio (see "A3" in Figure 8C). In this example, this horizontal image expansion is also performed equally in both the left and right directions, as shown in the figure.

[0084] Let's look at a specific example. Suppose the image size of the training input image is 800 x 600 pixels with an aspect ratio of 4:3 (approximately 1.33), while the image size of the image to be inferred is 1600 x 900 pixels with an aspect ratio of 16:9 (approximately 1.78). In this case, first, the image is expanded in all directions (up, down, left, and right) to obtain an expanded image with an image size of 1680 x 945 pixels. The aspect ratio of this expanded image is approximately 1.78. Therefore, in this case, the current aspect ratio (approximately 1.78) > target aspect ratio (approximately 1.33), and vertical image expansion is performed as shown in Figure 8B. Specifically, with a width of 1680, the vertical size required to match the target aspect ratio is 1260. Therefore, the conversion control unit 56 instructs the AI ​​processing unit 55 to generate an image with 715 pixels in the vertical direction, thereby generating an enlarged image with a size of 1680 x 1260 and an aspect ratio of 4:3.

[0085] As described above, the image conversion unit 49 in this example performs image expansion in all directions: up, down, left, and right, which improves the accuracy of image recognition processing. In cases where a part of the target subject extends beyond the frame, any direction in which that part extends beyond the frame is possible. Therefore, by performing image expansion in all directions as described above, an expanded image including that part can be obtained regardless of whether a part of the target subject extends beyond the frame in any direction (up, down, left, or right). Consequently, the possibility of recognizing the target subject as the target subject can be increased regardless of whether a part of the target subject extends beyond the frame in any direction (up, down, left, or right), thereby improving the accuracy of inference.

[0086] Furthermore, the image conversion unit 49 in this example performs image expansion with equal expansion width in all directions (up, down, left, and right). This makes it possible to equalize the probability that a portion of the subject in the original image will be included in the expanded image in all directions (up, down, left, and right) if that portion extends beyond the frame. Therefore, regardless of whether a portion of the subject extends beyond the frame (up, down, left, or right), the probability of recognizing the subject as the subject will be equal, thereby improving the accuracy of inference.

[0087] In this example, the image conversion unit 49, specifically the conversion control unit 56, performs image size adjustment processing on the image after aspect ratio conversion by image augmentation to match the image size of the training input image. Specifically, the conversion control unit 56 in this example performs the image size adjustment processing on the image that has been converted to an aspect ratio that matches the target aspect ratio as described above. For example, in light of the above specific example, an image with an image size of 1680 x 1260 generated by the AI ​​processing unit 55 is adjusted to the same image size as the training input image, which is 800 x 600. In this example, the image size information of the training input image can be obtained from the training image information IL.

[0088] In this example, an image adjusted to the same image size as the training input image through the image size adjustment process described above is output to the preprocessing unit 47 as the output image of the image conversion unit 49.

[0089] By performing the image size adjustment process described above, the resizing rate to the input tensor size during inference can be made the same as during training. Therefore, it is possible to prevent a decrease in inference accuracy caused by a difference in resizing rate between training and inference.

[0090] Furthermore, the image size adjustment process described above can be performed not only when the aspect ratio of the captured image does not match the aspect ratio of the training input image, but also when they do match.

[0091] (1-5. Processing Procedure) Referring to the flowchart in Figure 9, a specific example of a processing procedure that the conversion control unit 56 should execute in order to realize the image conversion method as the first embodiment described above will be explained. The main entity executing the processing in Figure 9 is the CPU of the conversion control unit 56, and the CPU executes the processing shown in Figure 9 based on a program stored in a predetermined storage device such as the ROM of the conversion control unit 56. In the following explanation, the main entity executing the processing in Figure 9 will be referred to as the conversion control unit 56.

[0092] First, in step S101, the conversion control unit 56 waits for a conversion execution instruction from the sensor internal control unit 43. As mentioned above, in this example, the sensor internal control unit 43 determines whether the aspect ratio of the captured image matches the aspect ratio of the training input image. If the two aspect ratios do not match, the sensor internal control unit 43 issues an instruction to the image conversion unit 49 (conversion control unit 56) to perform an aspect ratio conversion.

[0093] In step S101, if the sensor control unit 43 issues a conversion execution instruction, the conversion control unit 56 issues an instruction to generate an omnidirectional expanded image in step S102. That is, as explained with reference to Figure 8A above, it issues an instruction to perform image expansion of the input captured image by a predetermined expansion ratio in each of the up, down, left, and right directions. As mentioned above, in this example, in order to add sensor noise and image distortion to the expanded image, in the image expansion instructions in step S102, and in steps S106 and S107 described later, the AI ​​processing unit 55 is instructed to provide information on the sensor noise level and image distortion level, along with the instructions on the expansion direction and expansion width.

[0094] In step S103, following step S102, the conversion control unit 56 sets the target aspect ratio based on the learning image information IL. That is, it sets the aspect ratio of the learning input image indicated by the learning image information IL as the target aspect ratio.

[0095] In step S104, following step S103, the conversion control unit 56 calculates the aspect ratio of the omnidirectional expanded image as the current aspect ratio. That is, it calculates the aspect ratio of the omnidirectional expanded image generated in response to the instruction in step S102 as the current aspect ratio.

[0096] In step S105, following step S104, the conversion control unit 56 determines whether the current aspect ratio is greater than the target aspect ratio. If a positive result is obtained in step S105, indicating that the current aspect ratio is greater than the target aspect ratio, the conversion control unit 56 proceeds to step S106 and instructs the AI ​​processing unit 55 to perform vertical expansion to match the target aspect ratio.

[0097] On the other hand, if step S105 yields a negative result indicating that the current aspect ratio is not greater than the target aspect ratio, the conversion control unit 56 proceeds to step S107 and instructs the AI ​​processing unit 55 to perform lateral expansion to match the target aspect ratio.

[0098] The conversion control unit 56 proceeds to step S108 depending on whether it has performed the processing in step S106 or S107. In step S108, the conversion control unit 56 performs a resizing process to the image size used during training. That is, it receives the extended image generated by the AI ​​processing unit 55 in accordance with the instructions in step S106 or S107, and resizes the image size of the extended image to the image size of the training input image identified from the training image information IL. As described above, the image after such resizing is output from the image conversion unit 49 to the preprocessing unit 47. Note that the image size adjustment process may also be performed by a signal processing unit outside the conversion control unit 56.

[0099] In step S109, following step S108, the conversion control unit 56 determines whether or not processing is complete, that is, whether or not predetermined processing completion conditions, which are defined as conditions for ending the series of processes shown in Figure 9, have been met. Here, processing completion conditions can include, for example, the end of the imaging operation or the issuance of a processing completion instruction from the sensor control unit 43. If, in step S109, the processing completion conditions have not been met and a negative result indicating that processing is not complete is obtained, the conversion control unit 56 determines in step S110 whether or not it is time to start processing the next frame, that is, whether or not it is the start timing for processing the next frame. If a negative result indicating that it is not time to start processing the next frame is obtained, the process returns to step S109.

[0100] If a positive result indicating that processing of the next frame has started is obtained in step S110, the conversion control unit 56 returns to step S102 described above. As a result, the processing from steps S102 to S108 is repeatedly executed for each frame of the captured image until it is determined in step S108 that the processing has ended.

[0101] If the conversion control unit 56 obtains a positive result indicating the completion of processing in step S109, it terminates the series of processes shown in Figure 9.

[0102] While the above examples show image augmentation in all directions (up, down, left, and right), it is also possible to perform image augmentation in only one of these directions. Depending on the application of the inference system using the image recognition AI model, the direction in which part of the target subject extends beyond the frame may be limited to either the up / down or left / right directions. By performing image augmentation in only one of these directions, as described above, the image can be augmented in the direction in which the target subject extends beyond the frame, depending on the application, thereby increasing the likelihood of recognizing the target subject as it is.

[0103] <2. Second Embodiment> The second embodiment is an example of applying aspect ratio transformation by image augmentation to the learning side. In the first embodiment, it was assumed that the image recognition AI model was trained using training input images with a unified aspect ratio. In other words, it was assumed that the image recognition AI model was trained in a manner that made it difficult to absorb differences in aspect ratio.

[0104] In contrast, the second embodiment is based on the concept of creating an image recognition AI model capable of absorbing differences in the aspect ratio of a subject by providing the learning model of the image recognition AI model with images of various aspect ratios as learning input images. When preparing learning input images of various aspect ratios, aspect ratio conversion by image augmentation, as described in the embodiment, is applied.

[0105] Figure 10 is a block diagram showing an example configuration of the learning device 60 as a second embodiment. In the following description, parts that are the same as those already described will be denoted by the same reference numerals and their description will be omitted.

[0106] As shown in the figure, the learning device 60 includes a pre-processing unit 47, a post-processing unit 48, a learner 61, a learning control unit 62, a memory unit 63, a display unit 64, and an image conversion unit 65. These pre-processing unit 47, post-processing unit 48, learner 61, learning control unit 62, memory unit 63, display unit 64, and image conversion unit 65 are connected to a bus 66, and are capable of communicating data with each other via the bus 66.

[0107] The learning device 61 is configured to have a neural network for realizing an image recognition AI model, as well as a control circuit for adjusting the parameters of the neural network based on training data.

[0108] The learning control unit 62 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and controls the operation of each part connected via the bus 66 to perform overall control of the learning device 60.

[0109] The memory unit 63 is a non-volatile storage device such as an HDD or flash memory device, and is used to store various types of data related to learning. As shown in the figure, the learning dataset Ds is stored in the memory unit 63. The learning dataset Ds includes data such as training input images and training data used for machine learning of an image recognition AI model.

[0110] The display unit 64 is configured to have a display panel such as an LCD panel or an organic EL panel, and displays various types of information based on instructions from the learning control unit 62.

[0111] The learning control unit 62 controls the operation of the learner 61, including controlling the execution of learning operations and controlling the AI ​​model after learning to perform inference processing for verification. When the learning control unit 62 has the AI ​​model after learning to perform inference processing for verification, it performs processing to display an image showing the inference result obtained by the post-processing unit 48 (an image obtained by drawing the output of the AI ​​model onto the original image) on the display unit 64.

[0112] The image conversion unit 65 performs aspect ratio conversion by image augmentation on the training input images. Specifically, based on instructions from the learning control unit 62, the image conversion unit 65 performs aspect ratio conversion by image augmentation on the training input images stored in the memory unit 63 as the training dataset Ds. Although not shown in the figures, the image conversion unit 65 in this example has an image inference AI model Me, similar to the image conversion unit 49 described in the first embodiment, and is configured to perform image augmentation using the image inference AI model Me.

[0113] In this example, the image conversion unit 65 generates multiple aspect ratio-converted images with different aspect ratios from the same input image based on instructions from the learning control unit 62. Specifically, during machine learning using the learner 61, the learning control unit 62 selects one training input image from the training dataset Ds and provides the selected training input image to the image conversion unit 65 as an input image. Then, it instructs the image conversion unit 65 to use multiple aspect ratios as target aspect ratios, and converts the input image to an image with those instructed aspect ratios. The learning control unit 62 performs this control for all training input images in the training dataset Ds. This ensures that images with different aspect ratios are used for training for each image prepared as a training input image.

[0114] In the learning device 60, under the control of the learning control unit 62, the image whose aspect ratio has been converted by the image conversion unit 65 as described above, and the original learning input image (the image that has not undergone aspect ratio conversion) are sequentially input to the preprocessing unit 47 for resizing, and the resized image is sequentially input to the learner 61.

[0115] By inputting the above-described data into the learning device 61, it becomes possible to generate a trained image recognition AI model that can absorb differences in the aspect ratio of the subject.

[0116] To clarify, it is not essential for the image conversion unit 65 to convert the input image to multiple aspect ratios when obtaining images with multiple aspect ratios. Assuming that the original image of the training input image is input to the preprocessing unit 47 as described above, if the image conversion unit 65 generates at least one aspect ratio converted image, multiple images with different aspect ratios of the same image can be provided as input to the preprocessing unit 47. By adopting a configuration in which the image conversion unit 65 converts the input image to multiple aspect ratios, as in this example, it becomes possible to have three or more variations in the aspect ratio of the training input image, thereby improving the image recognition AI model's ability to absorb differences in the aspect ratio of the subject and improving its inference performance.

[0117] In the above description, an example was given in which the image conversion unit 65 is provided in a device equipped with a learning device 61. However, it is also conceivable that the image conversion unit 65 may be implemented in a device separate from the device equipped with the learning device 61.

[0118] <3. Modifications> Although embodiments of the present technology have been described above, the embodiments are not limited to the specific examples described above, and various modified configurations can be adopted. For example, regarding the first embodiment, the above example shows that sensor noise is added and image distortion is applied by the image inference AI model Me, but it is also conceivable that this addition of sensor noise and image distortion is performed in an image processing unit provided separately from the image inference AI model Me.

[0119] Figure 11 shows an example of the configuration of the image conversion unit 49A as a modified example in which the addition of sensor noise and image distortion is performed by an image processing unit separate from the image inference AI model. In this case, the AI ​​processing unit 55 is provided with an image inference AI model MeA instead of the image inference AI model Me. The image inference AI model MeA is configured to output only the image of the extended portion, rather than outputting an extended image, that is, an image in which the extended portion obtained by inference is synthesized (added) to the input captured image.

[0120] In the image conversion unit 49A, the image processing unit 57 applies sensor noise and image distortion according to the noise and distortion information to the extended image output from the image inference AI model MeA. Here, it is conceivable that the noise addition and image distortion processing by the image processing unit 57 be performed using rule-based processing rather than AI processing. Alternatively, these noise addition and image distortion processing processes could be implemented using AI processing.

[0121] Furthermore, the image conversion unit 49A is provided with a synthesis unit 58. The synthesis unit 58 synthesizes the extended image, which has had sensor noise and image distortion added by the image processing unit 57, with the captured image given as input to the image inference AI model MeA, and outputs the synthesized image to the conversion control unit 56A.

[0122] The conversion control unit 56A performs the aforementioned image size adjustment process on the composite image output from the synthesis unit 58, and outputs the image after the image size adjustment process as the converted image to the preprocessing unit 47.

[0123] Thus, the addition of sensor noise and image distortion is not limited to being implemented as a function of an image inference AI model.

[0124] Furthermore, while the above examples illustrate image augmentation using AI processing, it is also conceivable that image augmentation can be achieved using rule-based processing. For example, one method could be to generate the augmented portion of the image based on an image that is a copy of a part of the input image. Alternatively, the augmented portion could be assigned by selecting an image that matches the input image from a pre-prepared set of template images.

[0125] Furthermore, while the above example illustrates a configuration in which the imaging device 1 performs inference using an image recognition model, it is also conceivable that a configuration is adopted in which a device other than the imaging device 1 performs inference using an image recognition model on images captured by the imaging device 1. In that case, the image conversion unit 49 may be provided in a device other than the imaging device 1.

[0126] <4. Summary of Embodiments> As described above, the image processing apparatus (imaging device 1 or learning device 60) as an embodiment includes an image conversion unit (49, 65, or 49A) that performs aspect ratio conversion by image augmentation on an image that is the target of recognition by the image recognition AI model, which is an AI model that performs image recognition processing, before it is resized to the input tensor size of the image recognition AI model. As can be understood from the above description, the image conversion method as an embodiment is applicable to both inference and learning. If aspect ratio conversion by image augmentation as described above is performed during inference, it becomes possible to adjust the aspect ratio of the target image to the aspect ratio of the learning input image of the AI ​​model before resizing, thereby preventing the aspect ratio of the subject in the image used as the input tensor of the AI ​​model from differing between learning and inference, and improving inference performance. Furthermore, by performing aspect ratio conversion through image augmentation as described above during training, it becomes possible to provide the image recognition AI model's learner with multiple images with different aspect ratios as training input images. This makes it possible to generate an AI model that can absorb differences in the aspect ratio of the subject, thereby improving inference performance. Thus, the image processing apparatus of this embodiment can improve the inference performance of the AI ​​model that performs image recognition. In addition, by employing aspect ratio conversion through image augmentation, aspect ratio conversion can be achieved by splicing in parts of the image that are outside the subject of inference and are considered unimportant for inference. This prevents a decrease in inference performance caused by aspect ratio conversion.

[0127] Furthermore, in the image processing apparatus (imaging apparatus 1) as an embodiment, the image conversion unit (49) performs aspect ratio conversion by image augmentation on the image targeted for recognition by the image recognition AI model as a trained AI model. As a result, even if an image with a different aspect ratio from the training input image used to train the image recognition AI model is input as the target image for image recognition processing, it is possible to prevent the aspect ratio of the subject in the image as the input tensor of the AI ​​model from changing between training and inference, thereby improving inference performance.

[0128] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit performs aspect ratio conversion, using the aspect ratio of the training input image used to train the image recognition AI model as the target aspect ratio. This prevents the aspect ratio of the subject from differing between training and inference. Consequently, inference performance can be improved.

[0129] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit performs image augmentation using an image inference AI model that infers images outside the frame based on the image content within the frame. As a result, even if a part of the target subject extends beyond the frame in the original image, the image inference AI model can infer that part to be included within the frame and obtain an augmented image (aspect ratio converted image). If an image in which a part of the target subject extends beyond the frame is input to the image recognition AI model, there is a risk that the target subject may not be correctly recognized as the target subject. However, by inputting an augmented image that includes that part, the possibility of recognizing the target subject as the target subject can be increased, thereby improving the accuracy of inference.

[0130] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit performs image expansion in all directions: up, down, left, and right. In cases where a part of the subject extends beyond the frame, any direction in which the part extends beyond the frame is conceivable. By performing image expansion in all directions as described above, an expanded image including the part of the subject is obtained regardless of whether the part extends beyond the frame in any direction (up, down, left, or right). Therefore, the possibility of recognizing the subject as the subject can be increased regardless of whether the part extends beyond the frame in any direction (up, down, left, or right), thereby improving the accuracy of inference.

[0131] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit performs image expansion with equal expansion width in all directions (up, down, left, and right). This makes it possible to equalize the probability that the protruding portion of the subject will be included in the expanded image in all directions (up, down, left, and right) when a portion of the subject extends beyond the frame in the original image. Therefore, it is possible to ensure that the probability of recognizing the subject as the subject is equal regardless of whether a portion of the subject extends beyond the frame in any direction (up, down, left, or right), thereby improving the accuracy of inference.

[0132] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit obtains an extended image in which noise based on sensor noise information indicating the nature of sensor noise of the image sensor that obtains the captured image targeted for recognition by the image recognition AI model is added to the extended image. Since noise caused by the sensor noise of the image sensor is superimposed on the captured image, by adding noise to the extended image as described above, the noise characteristics of the entire extended image can be made uniform, and it is possible to prevent a decrease in inference performance due to differences in noise characteristics between the extended image and the rest of the image.

[0133] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit obtains an extended image in which image distortion is applied to the extended portion image based on distortion information indicating the manner of optical distortion occurring in the imaging optical system used to obtain the captured image that the image recognition AI model is to recognize. Since distortion occurs in the captured image due to the optical distortion of the imaging optical system, by applying distortion to the extended portion image as described above, the distortion characteristics of the entire extended image can be made uniform, and inference performance can be prevented from degrading due to differences in the distortion characteristics of the extended portion image and the rest of the image.

[0134] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit performs image expansion in only one of the two directions: vertical or horizontal. Depending on the application of the inference system using the image recognition AI model, the direction in which a part of the target subject extends beyond the frame may be limited to either the vertical or horizontal direction. By performing image expansion in only one of the two directions, vertical or horizontal, as described above, the image can be expanded in the direction in which the target subject extends beyond the frame, depending on the application, thereby increasing the possibility of recognizing the target subject as the target subject.

[0135] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit performs image size adjustment processing on the image after aspect ratio conversion by image augmentation to match the image size of the training input image used to train the image recognition AI model. This makes it possible to make the resizing rate to the input tensor size during inference the same as during training, thereby preventing a decrease in inference accuracy caused by a difference in resizing rate between training and inference.

[0136] Furthermore, in the image processing device (learning device 60) as an embodiment, the image conversion unit (65) performs aspect ratio conversion by image augmentation on the training input image used for training the image recognition AI model as the training target. This makes it possible to obtain multiple images with different aspect ratios as training input images for the image recognition AI model, and by resizing these multiple images with different aspect ratios to the input tensor size and providing them as input to the image recognition AI model as the training target, it becomes possible to generate an AI model that can absorb differences in the aspect ratio of the subject. Therefore, the inference performance of the image recognition AI model can be improved.

[0137] Furthermore, in the image processing apparatus as an embodiment, the image conversion unit generates multiple aspect ratio converted images with different aspect ratios from the same input image. This makes it possible to have three or more variations in the aspect ratio of the training input image, thereby improving the image recognition AI model's ability to absorb differences in the aspect ratio of the subject, and thus improving inference performance.

[0138] The image processing method as an embodiment involves an image processing device that performs aspect ratio transformation by image augmentation on an image that is the target of recognition by an image recognition AI model, which is an AI model that performs image recognition processing, and before the image is resized to the input tensor size of the image recognition AI model. This image processing method can also obtain the same functions and effects as the image processing device as an embodiment described above.

[0139] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0140] <5. This Technology> This technology can also be configured as follows: (1) An image processing device comprising an image conversion unit that performs aspect ratio conversion by image augmentation on an image that is the target of recognition by an image recognition AI model, which is an AI model that performs image recognition processing, before the image is resized to the input tensor size of the image recognition AI model. (2) The image processing device according to (1), wherein the image conversion unit performs aspect ratio conversion by image augmentation on an image that is the target of recognition by the image recognition AI model as a trained AI model. (3) The image processing device according to (2), wherein the image conversion unit performs aspect ratio conversion with the aspect ratio of the training input image used to train the image recognition AI model as the target aspect ratio. (4) The image processing device according to (2) or (3), wherein the image conversion unit performs image augmentation using an image inference AI model that infers an image outside the frame based on the image content inside the frame as the image augmentation. (5) The image processing apparatus according to (4), wherein the image conversion unit performs the image expansion in all directions: up, down, left, and right. (6) The image processing apparatus according to (5), wherein the image conversion unit performs the image expansion with equal expansion widths in all directions: up, down, left, and right. (7) The image processing apparatus according to any one of (4) to (6), wherein the image conversion unit obtains an expanded image in which noise based on sensor noise information indicating the nature of sensor noise of the image sensor that obtains the captured image that the image recognition AI model is to recognize is added to the expanded image. (8) The image processing apparatus according to any one of (4) to (7), wherein the image conversion unit obtains an expanded image in which image distortion based on distortion information indicating the nature of optical distortion occurring in the imaging optical system for obtaining the captured image that the image recognition AI model is to recognize is added to the expanded image. (9) The image processing apparatus according to claim 4, wherein the image conversion unit performs the image expansion only in the up, down, left, or right direction. The image processing apparatus according to (4). (10) The image processing apparatus according to any one of (2) to (9), wherein the image conversion unit performs an image size adjustment process on the image after aspect ratio conversion by the image augmentation to match the image size with the training input image used for training the image recognition AI model.(11) The image processing apparatus according to (1), wherein the image conversion unit performs aspect ratio conversion by image augmentation on a training input image used for training the image recognition AI model as a training target. (12) The image processing apparatus according to (11), wherein the image conversion unit generates a plurality of aspect ratio converted images with different aspect ratios from the same input image. (13) An image processing method in which the image processing apparatus performs aspect ratio conversion by image augmentation on an image that is the target of recognition by an image recognition AI model, which is an AI model that performs image recognition processing, and before the image is resized to the input tensor size of the image recognition AI model.

[0141] 1 Imaging device 2 Server device 3 User terminal NT Network 10 Image sensor 11 Imaging optical system 41 Imaging unit 42 Image signal processing unit 43 Sensor internal control unit 44 AI processing unit 45 Memory unit 47 Pre-processing unit 48 Post-processing unit 49, 49A Image conversion unit 55 AI processing unit Me, MeA Image inference AI model 56, 56A Conversion control unit 60 Learning device 61 Learner 62 Learning control unit 63 Memory unit 64 Display unit 65 Image conversion unit 66 Bus Ds Learning dataset 57 Image processing unit 58 Synthesis unit

Claims

1. An image processing apparatus comprising an image transformation unit that performs aspect ratio transformation by image augmentation on an image that is the target of recognition by an image recognition AI model, which is an AI model that performs image recognition processing, and which has not been resized to the input tensor size of the image recognition AI model.

2. The image processing apparatus according to claim 1, wherein the image conversion unit performs aspect ratio conversion by image augmentation on an image that the image recognition AI model, as a trained AI model, is to be recognized.

3. The image processing apparatus according to claim 2, wherein the image conversion unit performs an aspect ratio conversion, with the aspect ratio of the training input image used for training the image recognition AI model set as the target aspect ratio.

4. The image processing apparatus according to claim 2, wherein the image conversion unit performs image augmentation using an image inference AI model that infers an image outside the frame based on the image content within the frame.

5. The image processing apparatus according to claim 4, wherein the image conversion unit performs the image expansion in all directions: up, down, left, and right.

6. The image processing apparatus according to claim 5, wherein the image conversion unit performs the image expansion with equal expansion widths in the upper, lower, left, and right directions.

7. The image processing apparatus according to claim 4, wherein the image conversion unit obtains an extended image in which noise based on sensor noise information indicating the nature of sensor noise of the image sensor that obtains the captured image targeted for recognition by the image recognition AI model is added to the extended image.

8. The image processing apparatus according to claim 4, wherein the image conversion unit obtains an extended image in which image distortion is applied to the extended portion image based on distortion information indicating the manner of optical distortion occurring in the imaging optical system for obtaining the captured image that the image recognition AI model is to recognize.

9. The image processing apparatus according to claim 4, wherein the image conversion unit performs the image expansion in either the vertical or horizontal direction.

10. The image processing apparatus according to claim 2, wherein the image conversion unit performs an image size adjustment process on the image after aspect ratio conversion by image augmentation to match the image size with the training input image used for training the image recognition AI model.

11. The image processing apparatus according to claim 1, wherein the image conversion unit performs aspect ratio conversion by image augmentation on a training input image used for training the image recognition AI model as a training target.

12. The image processing apparatus according to claim 11, wherein the image conversion unit generates a plurality of aspect ratio converted images with different aspect ratios from the same input image.

13. An image processing method in which an image processing device performs aspect ratio transformation by image augmentation on an image that is the target of recognition by an image recognition AI model, which is an AI model that performs image recognition processing, and before the image is resized to the input tensor size of the image recognition AI model.

Images