Processing method, processing device, and program

Abstract

The present invention provides a processing method and processing device for appropriately utilizing processed image data when the field of view is changed for image data that includes supplementary information regarding the position of a subject within the field of view. [Solution] The processing method of the present invention processes image data including image information of a first field of view image and ancillary information attached to the image, and includes a conversion step of converting the image information of the first field of view image to image information of a second field of view image rotated from the first field of view, and an output step of outputting new position information regarding the position of the subject area in the second field of view based on information regarding the rotation from the first field of view to the second field of view and position information regarding the position of the subject area in the first field of view when the conversion step is performed, wherein in the conversion step, the portion of the second field of view image other than the rotated image is interpolated with an interpolated image, and when the conversion step is performed, information regarding the interpolated image is added to the ancillary information of the image data of the second field of view image.

Description

Technical Field

[0001] The present invention relates to an image processing method, a processing apparatus, and a program.

Background Art

[0002] Image data of a captured image is used for various purposes. For example, teacher data may be created from the image data and machine learning may be performed using the teacher data. The image data used for creating teacher data usually includes information (image information) of an image of an angle of view including a subject and a label (additional information) indicating the type of the subject and the area of the subject within the angle of view (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In addition, when using image data, the image may be processed, specifically, the angle of view may be changed by image conversion such as rotation and partial removal (trimming). In that case, it is necessary to appropriately obtain information regarding the area of the subject in the changed angle of view according to the degree of the angle-of-view change. In particular, when using image data for creating teacher data, if the additional information included in the image data after the angle-of-view change is not information reflecting the angle-of-view change, there is a risk of affecting the learning result.

[0005] One embodiment of the present invention aims to provide a processing method and a processing apparatus for appropriately using processed image data when changing the angle of view of image data including additional information regarding the position of a subject within the angle of view.

Means for Solving the Problems

[0006] To achieve the above objective, one embodiment of the present invention is a processing method for processing image data including image information of an image of a first field of view including a subject and ancillary information attached to the image, the processing method comprising: a conversion step of converting the image information of an image of a first field of view into image information of an image of a second field of view rotated from the first field of view; and an output step of outputting new position information relating to the position of the subject's region in the second field of view based on information relating to the rotation from the first field of view to the second field of view and position information relating to the position of the subject's region in the first field of view when the conversion step has been performed.

[0007] Furthermore, the output process may be performed when the rotation angle from the first field of view to the second field of view satisfies predetermined conditions. Furthermore, in the output process, a converted region may be set in the second field of view that is larger than the region of the subject in the first field of view and encompasses the region of the subject, and positional information relating to the converted region in the second field of view may be output as new positional information. Furthermore, the specified conditions are that the rotation angle is within a rotation range greater than 0° and less than 360°, excluding the rotation ranges of 90°, 180°, and 270°.

[0008] Furthermore, if the conversion process is performed, the process may further include a first additional step of adding information regarding rotation from the first to the second field of view to the supplementary information of the image data, which includes image information of the second field of view image. Furthermore, the conversion process may include an interpolation process in which the portion of the second field of view image other than the rotated image obtained by rotating the first field of view image is interpolated using an interpolated image. In addition, if the conversion process is performed, a second addition process may be included in which information about the interpolated image is added to the supplementary information of the image data containing the image information of the second field of view image.

[0009] Furthermore, image data may be used to create training data for machine learning.

[0010] Furthermore, an apparatus according to one embodiment of the present invention is an apparatus for processing image data including image information of an image of a first field of view including a subject and ancillary information attached to the image, and is configured to perform a conversion step of converting the image information of an image of a first field of view into image information of an image of a second field of view rotated from the first field of view, and an output step of outputting new position information relating to the position of the subject's region in the second field of view based on information relating to the rotation from the first field of view to the second field of view and position information relating to the position of the subject's region in the first field of view when the conversion step is performed.

[0011] Furthermore, a program according to one embodiment of the present invention is a program that causes a computer to perform the processing method according to one embodiment of the present invention described above.

[0012] Furthermore, another embodiment of the present invention is a processing method for processing image data including image information of an image with a first field of view that includes a subject, and ancillary information attached to the image, the method comprising: a conversion step of converting the image information of an image with a first field of view into image information of an image with a second field of view that is narrower than the first field of view; and an update step of updating the ancillary information when the subject area including the target subject, which is the subject in the first field of view, is removed by a predetermined percentage or more in the second field of view by the conversion step.

[0013] Furthermore, during the update process, the following steps may be performed: deleting information related to the attributes of the target subject from the supplementary information, or adding information related to the degree to which the subject area in the second field of view remains to the supplementary information. Furthermore, information regarding the degree to which the subject area remains in the second field of view may include the ratio of the remaining subject area in the second field of view to the subject area in the first field of view, or the number of pixels constituting the subject area in the second field of view. Furthermore, information regarding the degree of preservation of the subject area in the second field of view should be referenced when selecting image data that meets the set conditions in terms of preservation from an image database containing multiple image data.

[0014] Alternatively, the predetermined ratio may be set according to the attributes of the target subject. Also, when the target subject is a person, the predetermined ratio may vary according to the orientation of the person's face.

[0015] The subject area may be an area specified by the coordinates of a plurality of points in the subject area, or an area specified by the coordinates of a reference point in the subject area and the distance from the reference point.

[0016] Furthermore, when the conversion process is executed, a storage process may be further included for storing pre-conversion image data including the image information of the image with the first angle of view before conversion and the supplementary information regarding the position of the second angle of view at the first angle of view.

[0017] Moreover, a processing device according to an embodiment of the present invention is a processing device that processes image data including the image information of an image with a first angle of view including a subject and the supplementary information attached to the image, and includes a conversion process for converting the image information of the image with the first angle of view into the image information of an image with a second angle of view narrower than the first angle of view, and an update process for updating the supplementary information when a subject area including a target subject that is a subject at the first angle of view is removed by a predetermined ratio or more at the second angle of view by the conversion process.

[0018] Also, a program according to an embodiment of the present invention is a program for causing a computer to execute the processing method according to other embodiments of the present invention described above.

Brief Description of the Drawings

[0019] [Figure 1] It is an explanatory diagram of image data. [Figure 2] It is an explanatory diagram of the angle of view. [Figure 3] It is an explanatory diagram of the position information of a circular subject area. [Figure 4] It is an explanatory diagram of the rotation process. [Figure 5] It is an explanatory diagram of the removal process. [Figure 6] It is a configuration diagram of a processing system including a processing device according to one embodiment of the present invention. [Figure 7] It is a diagram showing the functions of a processing device according to one embodiment of the present invention. [Figure 8] It is a diagram showing image data before and after rotation processing. [Figure 9] It is a diagram showing image data before and after rotation processing and information regarding the position of the converted area. [Figure 10] It is a diagram showing image data before and after removal processing. [Figure 11] It is an explanatory diagram of the rotation processing flow. [Figure 12] It is an explanatory diagram of the removal processing flow. [Figure 13] It is an explanatory diagram of the removal processing according to the second embodiment of the present invention, and is a diagram showing image data before and after the removal processing (Part 1). [Figure 14] It is an explanatory diagram of the removal processing according to the second embodiment of the present invention, and is a diagram showing image data before and after the removal processing (Part 2).

Modes for Carrying Out the Invention

[0020] Specific embodiments of the present invention will be described. However, the embodiments described below are merely examples for facilitating the understanding of the present invention and do not limit the present invention. The present invention can be changed or improved from the embodiments described below without departing from its gist. Also, equivalents of the present invention are included therein.

[0021] In addition, in this specification, the concept of "device" includes a single device that exhibits a specific function, as well as a combination of a plurality of devices that are distributed and exist independently of each other while collaborating (cooperating) to exhibit a specific function.

[0022] Furthermore, in this specification, "person" refers to an entity that performs a specific action, and this concept includes individuals, groups, corporations and other legal entities, and organizations, and may even include computers and devices that constitute artificial intelligence (AI). Artificial intelligence is a system that realizes intelligent functions such as reasoning, prediction, and judgment using hardware and software resources. The algorithms of artificial intelligence are arbitrary and include, for example, expert systems, case-based reasoning (CBR), Bayesian networks, or inclusion architectures.

[0023] <<Summary of the First Embodiment of the Present Invention>> A first embodiment of the present invention relates to a processing method, processing apparatus, and program for processing image data generated by a known photographic device such as a digital camera.

[0024] As shown in Figure 1, image data includes image information of the captured image and associated information. The image is a field of view image containing one or more subjects. The image information includes information such as resolution and grayscale (2 colors) or RGB (Red, Green, Blue) color gradation values ​​for each pixel.

[0025] The field of view, as shown in Figure 2, is the data processing range in which an image is displayed or drawn, and is defined, for example, in a two-dimensional coordinate space with two mutually orthogonal axes forming a rectangle. Furthermore, as shown in Figure 2, the field of view is divided into multiple pixel regions aligned along each of the two coordinate axes. In this specification, a "pixel" is the smallest unit that constitutes an image, and a "pixel region" is a unit region corresponding to one pixel within the field of view. Hereafter, the number of pixel regions will be conveniently referred to as the "number of pixels".

[0026] The supplementary information includes tag information related to the image and its use, including so-called Exif (Exchangeable Image File Format) tag information. The supplementary information includes at least attribute information related to the attributes of the subject in the image and positional information related to the position of the subject's area within the field of view (hereinafter referred to as the subject area).

[0027] As shown in Figure 1, attribute information includes information indicating the type (category) of the subject when it is classified based on its characteristics or content, such as "female face" or "Japanese building." Attribute information may also include attributes other than type, such as the state of the subject, the presence or absence of blur or blur in the image, resolution, brightness, and the distance (depth) from the camera at the time of shooting.

[0028] As shown in Figure 1, the positional information is coordinate information representing the position of the subject area in the two-dimensional coordinate space that defines the field of view. The subject area is the region that encloses part or all of the subject, as shown in Figures 1 and 2. The shape of the subject area is not particularly limited, but may be, for example, roughly circular or rectangular. Furthermore, the "subject" is the object that is photographed by the camera, and is not limited to specific tangible objects, but also includes intangible objects such as landscapes or patterns.

[0029] The subject area is defined by the coordinates of multiple points within the subject area, or more precisely, multiple points located on the edges of the subject area. For example, if the subject area is rectangular, it is defined by the coordinates of two points located at the ends of the diagonal on the edge of the subject area (the points shown as white and black circles in Figures 1 and 2). In this case, the coordinates of the two identified points represent the positional information of the subject area. By using the coordinates of multiple points within the subject area in this way, the position of the subject area can be accurately represented.

[0030] Furthermore, the subject area may be defined by the coordinates of a base point within the subject area and the distance from that base point. For example, if the subject area is circular, as shown in Figure 3, the subject area is defined by the coordinates of the center (base point) of the subject area and the distance from the base point to the edge of the subject area (i.e., radius r). In this case, the coordinates of the center, which is the base point, and the radius, which is the distance from the base point, become the positional information of the subject area. In this way, the position of the subject area can be accurately represented using the base point within the subject area and the distance from the base point. Also, for example, if the subject area is rectangular, the subject area may be defined by the coordinates of the center of the subject area and the distance from the center in each coordinate axis direction (vertical and horizontal directions).

[0031] Furthermore, if multiple subjects are present within the field of view, the subject area is identified for each subject, and the aforementioned attribute information and location information of the subject area are generated for each subject. This attribute information and location information for each subject are then included in the supplementary information.

[0032] The supplementary information may be automatically added by the function of the camera (the function for adding supplementary information) when an image is taken with the camera. Alternatively, the supplementary information may be added by the function of an image editing application installed on the user-side device 12, which will be described later.

[0033] Image data is used for a variety of purposes, for example, to create training data for machine learning. More specifically, image data, including image information and associated information, is collected at a designated collection site, then annotated (selected) according to the learning objective, and training data is created from the selected image data. Once the necessary amount of training data for machine learning has been obtained, machine learning is performed using that training data.

[0034] Incidentally, for the purpose of effectively utilizing image data, image information may be transformed using known image editing techniques. Specifically, the image information of the image before transformation may be transformed into the image information of an image with a different field of view than the original field of view. Here, the field of view before transformation will be called the first field of view G1, and the field of view after transformation will be called the second field of view G2 (see Figures 4 and 5).

[0035] The first field of view is the field of view that reflects the field of view at the time of image capture. The field of view at the time of capture is determined according to the camera specifications, as well as the way the camera is held and its tilt relative to the horizontal at the time of capture. Furthermore, the origin of the coordinate space that defines the first field of view may be set at a predetermined position in the first field of view, for example, one of the vertices in the first field of view, or the center of the first field of view (i.e., the intersection of the diagonals).

[0036] The process of changing from the first field of view to the second field of view includes the rotation process of the affine transformation and the removal process that removes (trimmes) a portion of the first field of view. The rotation process, as shown in Figure 4, is the process of converting the image information of the first field of view G1 image into image information of the second field of view G2, which is obtained by rotating from the first field of view. The removal process, as shown in Figure 5, is a process that converts the image information of the first field of view G1 image to the image information of the second field of view G2 image, which is narrower than the first field of view. The converted image shown in Figure 4, as well as the rotation and removal processes, will be explained in detail later.

[0037] When image information is converted, the position of the subject area within the field of view changes. Therefore, when using the converted image data, it is necessary to consider the change in the position of the subject area. In this invention, taking this into consideration, a process (specifically, the output process or update process described later) is performed on the converted image data to provide information regarding the position of the subject area within the changed field of view. This allows the converted image data to be used appropriately and correctly.

[0038] <<Regarding the processing apparatus according to the first embodiment of the present invention>> As shown in Figure 6, the processing device (hereinafter referred to as the processing device 10) according to the first embodiment of the present invention constitutes a processing system S together with the user-side device 12 and the learning device 14. The processing system S performs machine learning according to the learning objective specified by the user, and provides the user with an inference model obtained as a result of the learning. By using the inference model, the user can determine or predict the type and state of the subject in the image.

[0039] The user-side device 12 consists of, for example, a PC (Personal Computer), a communication terminal, or a camera with communication capabilities owned by the user. The user-side device 12 provides image data of the captured image to the processing device 10, and more specifically, transmits the image data to the processing device 10 via the network N. The user-side device 12 provides image data that includes image information and associated information. Furthermore, the image data acquired by the processing unit 10 from the user-side device 12 may not contain any supplementary information at that time, and the processing unit 10 may add supplementary information to the image data after acquiring it.

[0040] The processing unit 10 is a computer and is communicatively connected to the user-side device 12 and the learning device 14. The processing unit 10 acquires image data from the user-side device 12 and performs a series of data processing operations on the image data, including the image information conversion described above (specifically, changing the field of view).

[0041] Furthermore, the processing unit 10 selects image data that meets predetermined conditions from a large number of image data, including image data before conversion and image data after conversion, by annotation, and creates training data for machine learning from the selected image data.

[0042] Furthermore, the process of creating training data from selected image data may be performed by a device other than the processing unit 10. Also, the annotation (selection) of image data may be performed before the conversion of image information on the image data. In that case, for reasons such as increasing the number of training data, the image information conversion (i.e., field of view change) may be performed on the image data selected by annotation.

[0043] Furthermore, the converted image data may be transmitted from the processing unit 10 to the user-side device 12. Upon receiving the converted image data, the user-side device 12 displays an image based on the image information contained in that image data, i.e., an image of the second field of view, on the display of the user-side device 12.

[0044] If the second field of view exceeds the size that can be displayed on the user-side device 12's display, the processing unit 10 sets the range of the second field of view to be displayed on the user-side device 12's display according to the size of the display. Similarly, if the aspect ratio of the second field of view does not match the aspect ratio of the display, the processing unit 10 sets the range of the second field of view to be displayed on the user-side device 12's display according to the aspect ratio of the display. The processing unit 10 then adds the information of the set display range to the supplementary information of the image data transmitted to the user-side device 12.

[0045] The learning device 14 is comprised of a computer and performs machine learning using training data created by the processing device 10. Machine learning is a technology and analytical technique related to artificial intelligence that learns patterns and criteria from data to predict and judge unknown events based on them. For example, machine learning is performed to identify the correspondence between the image features shown in image data that serve as training data and the accompanying information contained in the image data. More specifically, it learns the correspondence between the features of subjects in an image and the types of subjects indicated by the accompanying information, and then constructs an inference model that determines or predicts the types of subjects in an image based on the learning results.

[0046] The inference models constructed by machine learning can be any mathematical model, and examples of usable models include neural networks, convolutional neural networks, recurrent neural networks, attention models, transformers, generative adversarial networks, deep learning neural networks, Boltzmann machines, matrix factoryization, factoryization machines, MWAY factoryization machines, field-aware factoryization machines, field-aware neural factoryization machines, support vector machines, Bayesian networks, decision trees, and random forests.

[0047] In the configuration shown in Figure 6, the processing unit 10 and the learning device 14 exist independently of each other as separate devices, but the system is not limited to this configuration and may be integrated into a single device. Furthermore, the user-side device 12 may be equipped with some or all of the functions of the processing unit 10.

[0048] To illustrate an example of the configuration of the processing unit 10, the processing unit 10 is implemented by a processor and a program that the processor can execute, and is composed of, for example, a general-purpose computer. As shown in Figure 6, the computer that constitutes the processing unit 10 includes a processor 10A, memory 10B, and a communication interface 10C, etc.

[0049] The processor 10A is composed of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or a TPU (Tensor Processing Unit). Memory 10B is composed of semiconductor memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The communication interface 10C is configured, for example, by a network interface card or a communication interface board.

[0050] The computer comprising the processing unit 10 has a program installed for performing a series of processes on image data (hereinafter referred to as the image processing program). The image processing program is a program that causes the computer to implement the processing method of the present invention. In other words, when the processor 10A reads and executes the image processing program, the computer equipped with the processor 10A functions as the processing unit of the present invention.

[0051] The image processing program may be obtained by reading it from a computer-readable storage medium, or by receiving (downloading) it via a communication network such as the Internet or an intranet.

[0052] As shown in Figure 6, the processing system S is equipped with a storage device 16 that stores image data acquired by the processing device 10 from the user-side device 12. Multiple image data are stored in the storage device 16 as an image database DB, and the processing device 10 is configured to read the image data stored in the storage device 16. In addition, the image data stored in the image database DB includes image data whose image information has been converted by the processing device 10.

[0053] Furthermore, the image data stored in the storage device 16 may include image data obtained by scanning and digitizing printed (developed) analog photographs. The source of the image data is not particularly limited and may be other devices (not shown) other than the user-side device 12, for example, other devices (not shown) located on the network to which the processing device 10 is connected. Furthermore, the storage device 16 may be installed within the processing unit 10, or it may be provided on a third computer (e.g., an external server) that can communicate with the processing unit 10.

[0054] To explain the configuration of the processing unit 10 from a functional standpoint, as shown in Figure 7, the processing unit 10 includes an acquisition unit 21, a first conversion unit 22, an output unit 23, a first addition unit 24, a second addition unit 25, a second conversion unit 26, an update unit 27, and a storage unit 28. These functional units are realized through the cooperation of hardware equipment provided by the computer constituting the processing unit 10 and software including the aforementioned image processing program.

[0055] The acquisition unit 21 acquires image data from the user's device 12, etc. The acquired image data is stored in the storage device 16 and accumulated in the image database DB.

[0056] The first conversion unit 22 reads image data from the storage device 16 and performs a rotation process as the first conversion process on the image data. The amount of rotation from the first field of view G1 to the second field of view G2 in the rotation process, i.e., the rotation angle, is set within a rotation range greater than 0° and less than 360°. The second field of view G2 defined by the rotation process is defined as the range surrounding the first field of view G1x after rotation, as shown in Figure 4, more specifically, as a rectangular range tangent to the first field of view G1x after rotation.

[0057] The rotation angle in the rotation process can be arbitrarily determined within the above rotation range, and for example, the processing unit 10 may set the rotation angle randomly. Alternatively, the user may specify the rotation angle through the user-side device 12. In Figure 4, the rotation angle is 45°.

[0058] Furthermore, if the rotation angle satisfies predetermined conditions, specifically if the rotation angle falls within the rotation range excluding 90°, 180°, and 270°, the size of the second field of view will be the area surrounding the first field of view G1x. Therefore, in this case, the size of the second field of view will be larger than the size of the first field of view. In this case, as shown in Figure 4, the image P2 of the second field of view includes the rotated image RP obtained by rotating the image P1 of the first field of view, and the four sides of the rotated image RP and the adjacent interpolated image HP.

[0059] The interpolated image HP is a pseudo-image that interpolates the portion of the second-angle image P2 other than the rotated image RP. For example, it may be an image filled with one or more colors, or an image with a predetermined pattern. The color scheme of the interpolated image HP is not particularly limited, but as shown in Figure 4, the color of the interpolated image HP may be set to the same color as the pixels of the adjacent edges of the four sides of the rotated image RP.

[0060] Furthermore, depending on the rotation angle, the aspect ratio of the second field of view G2 may differ from the aspect ratio of the first field of view. The image data that has undergone rotation processing is then stored in the storage device 16 and accumulated in the image database DB.

[0061] The output unit 23 outputs new position information regarding the position of the subject area in the second field of view G2 when rotation processing is performed. "New position information" refers to, for example, position information regarding the position of the transformed area Ax in the second field of view G2. Here, the position of the transformed area Ax in the second field of view is represented in the coordinate space that defines the second field of view G2.

[0062] As shown in Figure 4, the converted region Ax is a region in the second field of view G2 that is larger than the subject region Ar in the first field of view (indicated by a dashed line in the figure) and encompasses the subject region Ar (indicated by a double dashed line in the figure). More specifically, it is the region tangent to the subject region Ar. The converted region Ax is set by the output unit 23 when the rotation process is performed.

[0063] The new position information, that is, the position information for the transformed region Ax, is output based on information regarding the rotation from the first field of view to the second field of view (hereinafter referred to as rotation information) and position information regarding the position of the subject region A3 in the first field of view before rotation processing. Here, the position of the subject region Ar in the first field of view before rotation processing is represented in the coordinate space that defines the first field of view G1 before rotation processing.

[0064] One way to output new position information is, for example, as shown in Figure 8, to replace the position information of the subject area in the supplementary information of the image data after rotation with the position information of the transformed area Ax. Alternatively, as shown in Figure 9, the position information of the subject area Ar before rotation may be included in the supplementary information of the image data after rotation, along with the position information of the transformed area Ax. The image data that has undergone rotation processing is the image data that includes the image information of the second field of view image P2, and will be referred to as the image data after rotation processing below.

[0065] Furthermore, as mentioned above, the display of the user-side device 12 can display the image P2 of the second field of view. On the other hand, if the rotation angle in the rotation process satisfies predetermined conditions, the second field of view G2 may become larger than the first field of view G1, and as a result, the second field of view G2 may exceed the size that can be displayed on the user-side device 12. In that case, the output unit 23 may set the range to be displayed on the user-side device 12 in the second field of view G2 and output the information of the set display range along with the position information of the converted area Ax.

[0066] The first addition unit 24 adds information related to the rotation process to the associated information contained in the image data after the rotation process, when the rotation process is performed. The information related to the rotation process is information related to the rotation from the first field of view to the second field of view, and specifically includes information such as the rotation angle, the rotation center, and the execution history of the rotation process. Furthermore, the information regarding the rotation process does not need to be the same as the rotation information referenced by the output unit 23 when outputting new position information.

[0067] The second addition unit 25 adds information about the complementary image HP to the supplementary information included in the image data after rotation processing, when rotation processing is performed and the rotation angle satisfies predetermined conditions. The information about the complementary image HP includes information about the position of the region of the complementary image HP in the second field of view G2, and information indicating that the image located in that region is the complementary image HP.

[0068] The second conversion unit 26 reads image data from the storage device 16 and performs a removal process as a second conversion process on the image data. The degree of removal from the first field of view G1 to the second field of view G2 in the removal process is set within a range greater than 0 and less than 100, where the number of pixels in the first field of view is 100.

[0069] In the removal process, the second field of view G2 is a rectangular area, and is set such that, for example, part or all of at least one subject (e.g., the main subject) included in the first field of view G1 remains within the second field of view G2. In other words, in the removal process, as shown in Figure 5, the area of ​​the first field of view G1 that is outside the second field of view G2 is removed.

[0070] The update unit 27 executes the first update process when the subject area including the target subject in the first field of view G1 is removed by a predetermined percentage or more in the second field of view G2 by the removal process. The target subject is a subject included in the first field of view whose subject area is removed by a predetermined percentage or more in the second field of view by the removal process. The first update process is a process of updating the ancillary information contained in the image data on which the removal process has been performed, and specifically, it is a process of deleting information related to the attributes of the target subject from the ancillary information.

[0071] To illustrate with an example from the case shown in Figure 10, suppose the first field of view G1 of the image data before the removal process contains four subjects, and the type (category) of each subject is "person," "dog," "tree," and "cloud." If, as a result of the removal process, the subject areas of "dog," "tree," and "cloud" are removed by a predetermined percentage or more in the second field of view G2, then three subjects, "dog," "tree," and "cloud," become the target subjects. In this case, the first update process is executed, and as a result, the attribute information of the three target subjects mentioned above is deleted, as shown in Figure 10.

[0072] If the first field of view G1 of the image data before the removal process includes multiple subjects, the update unit 27 calculates the removal ratio in the second field of view G1 for each subject area using the following formula (1). Removal ratio in the second field of view = (k1-k2) / k1 × 100 Equation (1) In equation (1), k1 represents the number of pixels in each subject area in the first field of view G1, and k2 represents the number of pixels in each subject area in the second field of view G2. For example, if the number of pixels in the subject area of ​​"dog" in the first field of view G1 is 2000, and the removal process reduces the number of pixels in the subject area of ​​"dog" in the second field of view to 500, the removal ratio is 75 (= 1500 / 2000 × 100).

[0073] Furthermore, the update unit 27 determines whether the calculated removal rate for each subject area is equal to or greater than a predetermined rate, and sets subjects included in subject areas where the removal rate is equal to or greater than the predetermined rate as target subjects. Here, the predetermined rate that serves as the criterion for determining whether or not a subject is a target subject is set according to the attributes of the target subject (specifically, type, etc.), and may differ for each attribute. In this case, it is possible to appropriately determine whether or not a subject should be a target subject (in other words, whether or not it should be subject to the first update process) according to the attributes of the subject.

[0074] Furthermore, if the subject is a person, the above-mentioned predetermined ratio may differ depending on the direction of the person's face. It is possible to appropriately determine whether or not the person should be included as the subject (in other words, whether or not they should be included in the first update process) depending on the direction of their face.

[0075] Then, if a target subject exists, the update unit 27 executes a first update process. In the first update process, as shown in Figure 10, attribute information of the target subject is deleted from the supplementary information contained in the image data after the removal process. Here, the image data after the removal process refers to the image data on which the removal process has been performed.

[0076] In addition, the removal process may extract a portion of the subject in the first field of view in the second field of view. For example, in the case shown in Figure 10, the upper body portion of the subject "male" is extracted by the removal process. When a portion of the subject is extracted by the removal process in this way, the update unit 27 may update the attribute information of the extracted portion of the subject to content corresponding to the extracted portion (in Figure 10, "upper body of male"), as shown in Figure 10.

[0077] When the removal process is executed, the storage unit 28 stores the pre-conversion image data in the storage device 16 and accumulates it in the image database DB. The pre-conversion image data is image data that includes image information of the first field of view image P1 before the removal process (pre-conversion) and supplementary information regarding the position (more specifically, relative position) of the second field of view G2 in the first field of view G1.

[0078] Furthermore, it is preferable that the pre-conversion image data be converted to lower-capacity data by, for example, reducing the image quality of the first field of view image P1, before being stored in the storage device 16. In this case, the data storage capacity can be reduced, and the pre-conversion image data can be easily read. Furthermore, it is preferable that the pre-conversion image data is stored included in the image data after the removal process, or stored in the storage device 16 in association with the image data after the removal process. In this case, it is easy to identify which image data is the original data of the image data after the removal process.

[0079] <<Regarding the image data processing procedure according to the first embodiment of the present invention>> Next, the image data processing flow using the processing apparatus 10 according to the first embodiment of the present invention will be described. In the processing flow described below, the image data is processed using the processing method of the present invention. In other words, each step in the processing flow described below corresponds to a component of the processing method of the present invention. The processing flow described below is merely an example, and unnecessary steps may be deleted, new steps added, or the execution order of steps rearranged, without departing from the spirit of the present invention.

[0080] The processing flow is initiated, for example, when the processing unit 10 receives an execution request sent from the user-side device 12. During the processing flow, the processing unit 10 acquires image data from the user-side device 12, etc., and stores the acquired image data in the storage device 16. An image database DB is constructed in the storage device 16 as the image data is accumulated.

[0081] The processing flow includes the rotation processing flow shown in Figure 11 and the removal processing flow shown in Figure 12. The rotation processing flow is the flow that performs rotation processing when converting image information. The removal processing flow is the flow that performs removal processing when converting image information. Each processing flow will be described below.

[0082] (Rotation processing flow) Each step (process) in the rotation processing flow is executed by the processor 10A of the processing unit 10. The processor 10A reads the image processing program and executes a series of data processing operations related to the rotation processing flow.

[0083] In the rotation processing flow, the processor 10A reads image data from the storage device 16 and performs rotation processing on the read image data (S001). This step S001 corresponds to the conversion step in the rotation processing flow. In step S001, the image information contained in the image data, that is, the image of the first field of view P1, is converted into image information of the second field of view P2, which is obtained by rotating from the first field of view.

[0084] Next, the processor 10A determines whether the rotation angle in the rotation process satisfies predetermined conditions, or more specifically, whether the rotation angle is greater than 0° and less than 360°, and within the rotation range excluding 90°, 180°, and 270° (S002).

[0085] If the rotation angle satisfies predetermined conditions, the processor 10A executes an output step (S003). In the output step, as shown in Figure 11, the processor 10A sets the converted region Ax in the second field of view G2 (S004). The converted region Ax is set based on the position information of the subject region in the first field of view G1 before rotation processing and information regarding the rotation from the first field of view G1 to the second field of view G2 (rotation information), and is tangent to the subject region Ar in the second field of view G2 (see Figure 4).

[0086] Furthermore, in the output process, the processor 10A performs a interpolation process, placing a pseudo-image, the interpolated image HP, adjacent to each of the four sides of the rotated image RP, which is obtained by rotating the image P1 of the first field of view (S005). As a result, an image P2 of the second field of view is generated, in which the parts other than the rotated image RP are interpolated by the interpolated image HP.

[0087] Furthermore, in the output process, the processor 10A outputs new position information regarding the position of the subject area in the second field of view G2, based on the rotation information and the position information of the subject area in the first field of view before rotation processing (S006). Specifically, the processor 10A outputs the position information of the converted area Ax set in step S004 as new position information.

[0088] Furthermore, when the processor 10A outputs new position information, it may overwrite the position information of the subject region included in the supplementary information of the rotated image data with the position information of the converted region Ax (see Figure 8). Alternatively, the position information of the converted region Ax may be associated with the rotated image data and stored in a separate data file from that image data.

[0089] As described above, by outputting new positional information for the subject area, even if the position of the subject area changes due to the angle of view change caused by rotation, the position of the changed subject area (specifically, the transformed area Ax) can be appropriately identified. This allows for an accurate understanding of the correspondence between the position of the subject area and the attributes of the subject contained within the subject area in the rotated image data. As a result, when performing machine learning using training data created from rotated image data, highly accurate learning results can be obtained.

[0090] Furthermore, the transformed region Ax is set to be larger than the subject region Ar in the first field of view in the second field of view G2, and to encompass the subject region Ar (see Figure 4). Since the transformed region Ax contains the same subject as the subject region Ar contained within the transformed region Ax, even if the position of the subject region moves due to rotation, the subject contained within that subject region can be appropriately detected.

[0091] Returning to the description of the rotation processing flow, when rotation processing is performed, the processor 10A executes the first and second addition steps (S007, S008). In the first addition step, the processor 10A adds information related to the rotation processing to the supplementary information of the image data after rotation processing.

[0092] By adding information about the rotation process in the first addition step, it becomes easy to identify that the image data to which that information has been added is image data after rotation. This makes it easy to understand, for example, that when performing machine learning using training data created from image data after rotation, that the training data was created from image data after rotation. Furthermore, it becomes possible to identify the correspondence between the rotation angle and rotation center indicated by the rotation information and the accuracy of machine learning using training data created from image data after rotation.

[0093] In the second addition step, the processor 10A adds information about the complementary image HP to the added information of the image data after rotation processing. By adding information about the complementary image HP, it is easy to determine whether or not the image data contains the complementary image HP. As a result, in the selection (annotation) of image data for creating training data, the presence or absence of the complementary image HP can be taken into consideration when selecting image data.

[0094] Furthermore, when performing machine learning using training data created from image data that includes complementary image HPs, it is possible to identify the presence or absence of complementary image HPs and their positions based on information about them. This prevents the model from mistakenly recognizing complementary image HPs as correct images during training.

[0095] The rotation process flow ends when the series of steps S001 to S008 described above are completed.

[0096] (Removal process flow) Each step (process) in the removal process flow is executed by the processor 10A. The processor 10A reads the image processing program and executes a series of data processing operations related to the removal process flow.

[0097] In the removal process flow, the processor 10A reads image data from the storage device 16 and performs a removal process on the read image data (S011). This step S011 corresponds to the conversion step in the removal process flow. In step S011, the image information contained in the image data, that is, the image information of the first field of view G1, is converted into image information of the second field of view G2, which is narrower than the first field of view.

[0098] Next, the processor 10A determines whether the subject area in the first field of view G1 has been removed by a predetermined percentage or more in the second field of view G2 through the removal process (i.e., the conversion process) (S012). If the first field of view G1 contains multiple subject areas, the processor 10A repeats the steps from S012 onward for each subject area.

[0099] If a subject area, i.e., the subject area of ​​the target subject, remains after removal by a predetermined percentage or more, the processor 10A executes an update step (S013). In the update step, the processor 10A updates the supplementary information of the image data after removal, specifically deleting the attribute information of the target subject from the supplementary information (see Figure 10).

[0100] Target subjects from which a predetermined percentage or more of the subject area has been removed are difficult to identify in the second field of view G2, or they exist outside the second field of view G2, making attribute information unnecessary. Therefore, by performing the update process as described above and deleting the attribute information of the target subject from the accompanying information, the accompanying information of the image data after the removal process can be updated to appropriate information according to the second field of view G2. Furthermore, when performing machine learning using training data created from the image data after the removal process, deleting the attribute information of the target subject reduces the influence of the target subject on the learning results. As a result, highly accurate learning results can be obtained.

[0101] Returning to the explanation of the removal process flow, when the processor 10A performs the removal process (i.e., the conversion process), it also performs a storage process (S014). In the storage process, the processor 10A stores the pre-conversion image data in the storage device 16.

[0102] As described above, by storing the image data before conversion, machine learning can be performed using training data created from the image data before conversion. This machine learning can identify the relationship between the first field of view P1 and the second field of view P2, or more simply, the trend regarding the position of the region containing the main subject in the first field of view P1. In other words, as a result of learning, a model can be constructed to detect the main subject in the image.

[0103] Furthermore, from the viewpoint of data storage capacity and data reading speed, it is preferable that the pre-conversion image data be converted to a lower-capacity data before being stored in the storage device 16. It is also preferable that the pre-conversion image data be stored in the storage device 16 in association with the image data after the removal process. In addition, the low-capacity pre-conversion image data may be recorded together with the converted image data as a thumbnail image. In this case, it is more preferable that the ancillary information of the image data after the removal process includes positional information of the subject area in the pre-conversion image data, similar to the image data shown in Figure 9.

[0104] The removal process flow ends when the series of steps S011 to S014 described above are completed.

[0105] <<Other Embodiments>> The embodiments described above are specific examples given to clearly illustrate the processing method, processing apparatus, and program of the present invention, and are merely examples; other embodiments may also be conceivable.

[0106] In the first embodiment described above, in the update step when the removal process is performed, attribute information of the target subject whose subject area has been removed by a predetermined percentage or more is deleted from the supplementary information of the image data after the removal process. However, the method for updating the supplementary information in the update step may be other than the method described above.

[0107] A second embodiment of the present invention, in which the update process is carried out by a method different from that described above, is described below. In the following, the configuration of the second embodiment that differs from that of the first embodiment will be mainly described, and the configuration that is common to both the first and second embodiments will not be described.

[0108] In the update step according to the second embodiment, the processor 10A performs a third addition step instead of deleting the attribute information of the target subject. The third addition step is a process of adding information regarding the degree to which the subject area of ​​the target subject in the second field of view remains (hereinafter referred to as "degree of remaining information") to the supplementary information of the image data from which the removal process has been performed.

[0109] The residual degree information is information that shows the residual percentage, for example, as shown in Figure 13. The residual percentage is the ratio of the subject area in the second field of view to the subject area in the first field of view for each subject, including the target subject. Specifically, when the removal percentage mentioned above is j, the residual percentage can be calculated by the following formula (2). For example, if the residual percentage is 100, it means that the target subject is completely remaining in the second field of view. On the other hand, if the residual percentage is 0, it means that the target subject has been completely removed in the second field of view. Remaining ratio=100-j (2)

[0110] Furthermore, the residual degree information is not limited to the residual percentage, but may also be the number of pixels constituting the subject area in the second field of view G2 (remaining pixels), as shown in Figure 14. In this case, residual degree information may be provided by setting the remaining pixels and the number of pixels in the subject area in the first field of view G1 (i.e., the subject area before the removal process). Alternatively, residual degree information may be provided by setting the remaining pixels and the number of pixels removed by the removal process.

[0111] Furthermore, information indicating whether the subject area remains within the second field of view G2, or in other words, whether the subject area is outside the second field of view G2, may be added as residual degree information. In addition, if the subject is a person, the residual degree information may include the proportion of the subject area remaining within the second field of view G2 that is occupied by the person's face.

[0112] As described above, by adding information on the degree of remaining subject area of ​​the target subject to the supplementary information of the image data after the removal process, it is possible to easily understand the extent to which the subject area of ​​the target subject has been removed during the removal process. Furthermore, by referring to the degree of remaining information, it is possible to determine whether the image data after the removal process is suitable for use.

[0113] Specifically, image data after the removal process may be used to create training data, provided that the numerical value indicated by the degree of remaining information (for example, the percentage of remaining pixels or the number of remaining pixels) is above a threshold. This is because the higher the above value, the easier it is to identify the target subject included in the subject area. Based on this, if training data is created using image data in which the above value has reached a predetermined standard (threshold), the reliability (credibility) of the training data will improve, resulting in more accurate machine learning.

[0114] On the other hand, image data after the removal process may be used to create training data, provided that the remaining percentage, remaining pixel count, or the proportion of a person's face in the subject area, as indicated by the remaining percentage information, is below a threshold. This is because, for example, if the subject is a person, using image data of that person clearly without permission could raise issues regarding their right to privacy. Taking this into consideration, by selecting image data in which the above values ​​do not reach a predetermined standard (threshold), it is possible to avoid issues concerning the rights of the subject.

[0115] As described above, the degree of preservation information is referenced when selecting image data from the image database DB that meet the set conditions (specifically, the condition that the degree of preservation of the subject area exceeds or falls below a threshold). By referring to the degree of preservation information, it is possible to select image data suitable for use.

[0116] Furthermore, in the second embodiment, the attribute information of the subject whose area has been removed by the removal process by a predetermined percentage or more, i.e., the target subject, is not deleted and remains in the supplementary information. Therefore, when performing machine learning using training data created from the image data after the removal process, the attribute information of the target subject can be used. This makes it possible to perform machine learning by utilizing not only the attribute information of the subject included in the second field of view G2, but also the attribute information of the subject outside the second field of view G2. As a result, for example, machine learning can be performed by taking into account the shooting location or scene of the original image indicated by the image data before the removal process, in other words, the background captured in the image P1 of the first field of view.

[0117] Furthermore, in the above-described embodiment, the image data from which the image information has been transformed (i.e., image data from which rotation or removal processing has been performed) is used to create training data for machine learning. However, the above-described image data may also be used for purposes other than creating training data.

[0118] Furthermore, the processor of the processing apparatus of the present invention may be provided in both the user-side device 12 and the server computer capable of communicating with the user-side device 12. Also, if there are multiple processors, these multiple processors may cooperate with each other.

[0119] The processor included in the processing apparatus of the present invention includes various types of processors. These various types of processors include, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. Furthermore, various types of processors include PLDs (Programmable Logic Devices), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing. Furthermore, various types of processors include dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform particular processing.

[0120] Furthermore, one functional unit of the processing apparatus of the present invention may be configured by one of the various processors described above, or by a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of FPGA and CPU. Furthermore, the multiple functional units of the processing apparatus of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be combined and configured by a single processor. Furthermore, as in the embodiment described above, a single processor may be configured by a combination of one or more CPUs and software, and this processor may function as multiple functional units.

[0121] Furthermore, the present invention may utilize a processor that realizes the functions of the entire system, including multiple functional units, on a single IC (Integrated Circuit) chip, such as a System on Chip (SoC). Additionally, the hardware configuration of the various processors described above may be an electrical circuit (Circuitry) combining circuit elements such as semiconductor devices. [Explanation of Symbols]

[0122] 10 Processing Unit 10A Processor 10B memory 10C communication interface 12. User-side equipment 14 Learning device 16 Storage device 21 Acquisition Department 22 First Conversion Unit 23 Output section 24. First Addition Section 25 Second Attachment 26 Second Conversion Unit 27 Update section 28 Memory section Ar subject area Ax transformed region G1 First angle of view G1x First field of view after rotation G2 Second field of view DB Image Database HP supplementary images N Network P1 Image of the first field of view P2 Image with the second field of view RP Rotating Image S Processing System

Claims

1. A processing method for image data that includes image information of an image of a first field of view including a subject, and supplementary information attached to the image, A conversion step that converts the image information of the image with the first field of view into the image information of the image with the second field of view obtained by rotating from the first field of view, When the conversion step is performed, the output step includes outputting new position information regarding the position of the subject's region in the second field of view, based on information regarding rotation from the first field of view to the second field of view and position information regarding the position of the subject's region in the first field of view. The conversion step includes an interpolation step in which the portion of the second field of view image other than the rotated image obtained by rotating the first field of view image is interpolated with an interpolated image. A processing method further comprising a second addition step of adding information about the complementary image to the ancillary information of the image data, which includes the image information of the second field of view, when the conversion step is performed.

2. The processing method according to claim 1, wherein the output step is performed when the rotation angle from the first angle of view to the second angle of view satisfies a predetermined condition.

3. The processing method according to claim 2, wherein in the output step, a converted region is set in the second field of view that is larger than the region of the subject in the first field of view and encompasses the region of the subject, and positional information relating to the converted region in the second field of view is output as the new positional information.

4. The processing method according to claim 2, wherein the predetermined condition is that the rotation angle is within a rotation range greater than 0° and less than 360°, excluding the rotation ranges of 90°, 180°, and 270°.

5. The processing method according to claim 1, further comprising a first addition step of adding information relating to rotation from the first angle of view to the second angle of view to the ancillary information of the image data, which includes image information of the image of the second angle of view, when the conversion step is performed.

6. The processing method according to claim 1, wherein the image data is used to create training data for machine learning.

7. A processing device for processing image data, which includes image information of an image of a first field of view including a subject, and supplementary information attached to the image, A conversion step that converts the image information of the image with the first field of view into the image information of the image with the second field of view obtained by rotating from the first field of view, When the conversion step is performed, an output step outputs new position information relating to the position of the subject's region in the second field of view, based on information relating to the rotation from the first field of view to the second field of view and position information relating to the position of the subject's region in the first field of view. It is configured to perform, The conversion step includes an interpolation step in which the portion of the second field of view image other than the rotated image obtained by rotating the first field of view image is interpolated with an interpolated image. A processing apparatus configured to further perform a second addition step, which, when the conversion step is performed, adds information about the complementary image to the ancillary information of the image data, which includes the image information of the second field of view image.

8. A program for causing a computer to perform the processing method described in claim 1.

Images