Processing method, processing device, and program

Abstract

Provided is a processing method and a processing device for suitably using processed image data when an angle of view has been modified in image data including incidental information relating to the position of a subject in the angle of view.　This processing method processes image data including image information about an image of a first angle of view including a subject and incidental information imparted to the image, the method including: a conversion step for converting the image information about an image of the first angle of view into image information about an image of a second angle of view rotated from the first angle of view; and an output step for outputting, when the conversion step is implemented, new position information relating to the position of a subject region in the second angle of view on the basis of information relating to rotation from the first angle of view to the second angle of view and position information relating to the position of a subject region in the first angle of view.

Description

[Technical Field]

[0001] The present invention relates to an image processing method, a processing device, and a program. [Background technology]

[0002] Image data of captured images is used for various purposes, for example, teacher data may be created from the image data and machine learning may be performed using the teacher data. Image data used to create the teacher data usually includes information (image information) of an image with an angle of view that includes a subject, and labels (accompanying information) that indicate the type of subject and the area of ​​the subject within the angle of view (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-18384 Summary of the Invention [Problem to be solved by the invention]

[0004] Furthermore, when using image data, the image may be processed, specifically, the angle of view may be changed by image transformation such as rotation and partial removal (cropping). In such cases, it is necessary to appropriately acquire information about the area of ​​the subject at the changed angle of view depending on the degree of the change in the angle of view. In particular, when image data is used to create training data, if the additional information included in the image data after the change in the angle of view does not reflect the change in the angle of view, there is a risk that it will affect the learning results.

[0005] One embodiment of the present invention aims to provide a processing method and processing device for appropriately utilizing processed image data when the angle of view of image data containing additional information regarding the position of a subject within the angle of view is changed. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, one embodiment of the present invention is a processing method for processing image data including image information of an image at a first angle of view that includes a subject and additional information assigned to the image, the processing method including: a conversion process for converting the image information of the image at the first angle of view into image information of an image at a second angle of view that is rotated from the first angle of view; and an output process for outputting, when the conversion process is executed, new position information regarding the position of the subject's area at the second angle of view based on information regarding the rotation from the first angle of view to the second angle of view and position information regarding the position of the subject's area at the first angle of view.

[0007] The output step may be executed when the rotation angle from the first angle of view to the second angle of view satisfies a predetermined condition. In addition, in the output process, a converted area may be set at the second angle of view that is larger than the area of ​​the subject at the first angle of view and includes the area of ​​the subject, and position information regarding the converted area at the second angle of view may be output as new position information. The predetermined condition may be that the rotation angle is within a rotation range of more than 0° and less than 360°, excluding 90°, 180°, and 270°.

[0008] In addition, when the conversion step is executed, the method may further include a first addition step of adding information regarding rotation from the first angle of view to the second angle of view to the auxiliary information of the image data including image information of the image with the second angle of view. The converting step may also include a complementing step of complementing, with a complement image, a portion of the image of the second angle of view other than the rotated image obtained by rotating the image of the first angle of view. Furthermore, the method may further include a second adding step of, when the converting step is executed, adding information about the complement image to incidental information of the image data including image information of the image of the second angle of view.

[0009] The image data may also be used to create training data for use in machine learning.

[0010] Furthermore, a processing device according to one embodiment of the present invention is a processing device that processes image data including image information of an image of a first angle of view that includes a subject and additional information attached to the image, and is configured to perform a conversion process that converts the image information of the image of the first angle of view into image information of an image of a second angle of view that is rotated from the first angle of view, and an output process that, when the conversion process is executed, outputs new position information regarding the position of the area of ​​the subject in the second angle of view based on information regarding the rotation from the first angle of view to the second angle of view and position information regarding the position of the area of ​​the subject in the first angle of view.

[0011] A program according to an embodiment of the present invention is a program for causing a computer to execute the processing method according to the embodiment of the present invention described above.

[0012] Furthermore, a processing method according to another embodiment of the present invention is a processing method for processing image data including image information of an image of a first angle of view including a subject and additional information assigned to the image, and includes a conversion process for converting the image information of the image of the first angle of view into image information of an image of a second angle of view narrower than the first angle of view, and an update process for updating the additional information when a subject area including a target subject that is a subject in the first angle of view is removed by a predetermined percentage or more in the second angle of view by the conversion process.

[0013] In addition, in the updating step, a process of deleting information relating to the attributes of the target subject from the supplementary information, or a process of adding information relating to the degree to which the subject area remains in the second angle of view to the supplementary information may be executed. Furthermore, the information regarding the degree to which the subject area remains in the second angle of view may include the remaining ratio of the subject area in the second angle of view to the subject area in the first angle of view, or the number of pixels that make up the subject area in the second angle of view. Furthermore, the information regarding the degree of remaining of the subject area at the second angle of view may be information that is referenced when selecting image data whose degree of remaining satisfies a set condition from an image database in which a plurality of image data has been accumulated.

[0014] The predetermined ratio may also be set according to the attributes of the target subject. Furthermore, when the target subject is a person, the predetermined ratio may differ depending on the direction of the person's face.

[0015] The subject area may also be an area specified by the coordinates of a plurality of points in the subject area, or an area specified by the coordinates of a base point in the subject area and the distance from the base point.

[0016] In addition, when the conversion process is executed, the method may further include a storage process for storing pre-conversion image data including image information of the image at the first angle of view before conversion and additional information regarding the position of the second angle of view within the first angle of view.

[0017] Furthermore, a processing device according to one embodiment of the present invention is a processing device that processes image data including image information of an image of a first angle of view that includes a subject and additional information that is assigned to the image, and is configured to execute a conversion process that converts the image information of the image of the first angle of view into image information of an image of a second angle of view that is narrower than the first angle of view, and an update process that updates the additional information when a subject area that includes a target subject that is a subject at the first angle of view is removed by more than a predetermined percentage at the second angle of view by the conversion process.

[0018] A program according to one embodiment of the present invention is a program for causing a computer to execute the processing method according to the other embodiment of the present invention described above. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 2 is an explanatory diagram of image data. [Figure 2] FIG. [Figure 3] FIG. 10 is an explanatory diagram of position information of a circular subject region. [Figure 4] FIG. 10 is an explanatory diagram of a rotation process. [Figure 5] FIG. 10 is an explanatory diagram of a removal process. [Figure 6] 1 is a configuration diagram of a processing system including a processing apparatus according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating the functionality of a processing device according to one embodiment of the present invention. [Figure 8] 10A and 10B are diagrams showing image data before and after rotation processing; [Figure 9] 10A and 10B are diagrams showing image data before and after rotation processing, and information relating to the position of the post-conversion area. [Figure 10] 10A and 10B are diagrams showing image data before and after removal processing. [Figure 11] FIG. 10 is an explanatory diagram of a rotation processing flow. [Figure 12] FIG. 10 is an explanatory diagram of a removal processing flow. [Figure 13] FIG. 10 is an explanatory diagram of the removal process according to the second embodiment of the present invention, showing image data before and after the removal process (part 1). [Figure 14] FIG. 10 is an explanatory diagram of the removal process according to the second embodiment of the present invention, showing image data before and after the removal process (part 2). DETAILED DESCRIPTION OF THE INVENTION

[0020] Specific embodiments of the present invention will be described. However, the embodiments described below are merely examples for facilitating understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0021] In addition, in this specification, the concept of "device" includes not only a single device that performs a specific function, but also a combination of multiple devices that exist independently and in a distributed manner but cooperate (link) to perform a specific function.

[0022] In this specification, the term "person" refers to an entity that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, as well as computers and devices that constitute artificial intelligence (AI). AI is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. AI algorithms are arbitrary, and examples include expert systems, case-based reasoning (CBR), Bayesian networks, and subsumption architectures.

[0023] <<Overview of the First Embodiment of the Present Invention>> A first embodiment of the present invention relates to a processing method, a processing device, and a program for processing image data generated by a known photographing device such as a digital camera.

[0024] As shown in Fig. 1, image data includes image information of a captured image and additional information added to the image. The image is an image with a field of view that includes one or more subjects. The image information is information that indicates, for each pixel, the resolution and grayscale values ​​of two colors (black and white) or three colors (RGB (Red, Green, Blue)).

[0025] As shown in Figure 2, the angle of view is a data processing range in which an image is displayed or drawn, and is defined in a two-dimensional coordinate space having, for example, a rectangular shape and two mutually perpendicular axes as coordinate axes. Also, as shown in Figure 2, the angle of view is divided into a plurality of pixel regions aligned along each of the two coordinate axes. In this specification, a "pixel" is the smallest unit constituting an image, and a "pixel region" is a unit region within the angle of view that corresponds to one pixel. Hereinafter, the number of pixel regions will be referred to as the "number of pixels" for convenience.

[0026] The incidental information is tag information related to the image and its use, etc., and includes tag information in the so-called Exchangeable Image File Format (Exif) format, etc. The incidental information includes at least attribute information related to the attributes of the subject in the image and position information related to the position of the subject area (hereinafter referred to as the subject area) in the angle of view.

[0027] The attribute information includes information indicating the type (category) of the subject when the subject is classified based on its characteristics or content, such as "female face" or "Japanese building," as shown in Fig. 1. Note that the attribute information may also include attributes other than the type, such as the state of the subject, the presence or absence of blur or shake in the image, resolution, brightness, and the distance (depth) from the camera when the image was taken.

[0028] The position information is coordinate information that indicates the position of the subject area in a two-dimensional coordinate space that defines the angle of view, as shown in Figure 1. The subject area is an area that surrounds part or all of the subject, as shown in Figures 1 and 2. The shape of the subject area is not particularly limited, and may be, for example, approximately circular or rectangular. Furthermore, the "subject" is an object photographed by a camera, and is not limited to a specific tangible object, but also includes non-tangible objects such as landscapes or patterns.

[0029] The subject area is an area specified by the coordinates of multiple points in the subject area, more specifically, multiple points located on the edge of the subject area. For example, if the subject area is rectangular, the subject area is specified by the coordinates of two points located at both ends of the diagonal line on the edge of the subject area (shown as white and black circles in Figures 1 and 2). In this case, the coordinates of the two specified points become the position information of the subject area. In this way, by using the coordinates of multiple points in the subject area, the position of the subject area can be accurately represented.

[0030] The subject area may also be an area specified by the coordinates of a base point within the subject area and the distance from the base point. For example, if the subject area is circular, as shown in FIG. 3, the subject area is specified by the coordinates of the center (base point) of the subject area and the distance from the base point to the edge of the subject area (i.e., radius r). In this case, the coordinates of the center, which is the base point, and the radius, which is the distance from the base point, are the position information of the subject area. In this way, by using the base point within the subject area and the distance from the base point, the position of the subject area can be accurately represented. For example, if the subject area is rectangular, the subject area may also be specified by the coordinates of the center of the subject area and the distance from the center in each coordinate axis direction (vertical and horizontal).

[0031] In addition, when there are multiple subjects within the field of view, the subject area is identified for each subject, the above-mentioned attribute information and position information of the subject area are generated for each subject, and the attribute information and position information for each subject are included in the supplementary information.

[0032] The incidental information may be automatically added by a function of the photographing device (function of adding incidental information) when the image is captured by the photographing device, or may be added by a function of an image editing application installed in the user-side device 12 (described later).

[0033] Image data is used for a variety of purposes, for example, to create training data for machine learning. More specifically, image data containing image information and additional information is collected at a predetermined collection point, then annotated (selected) according to the learning objective, and training data is created from the selected image data. Once the required amount of training data for machine learning is acquired, machine learning is performed using the training data.

[0034] Incidentally, image data may be converted using a known image editing technique in order to effectively utilize the image data. Specifically, image information of an image before conversion may be converted into image information of an image with a field of view changed from the field of view before conversion. Here, the field of view before conversion is referred to as a first field of view G1, and the field of view after conversion is referred to as a second field of view G2 (see FIGS. 4 and 5).

[0035] The first angle of view is an angle of view that reflects the angle of view when an image is captured. The angle of view when capturing an image is determined depending on the specifications of the camera, the state in which the camera is held when capturing an image, and the inclination relative to the horizontal direction. The origin of the coordinate space that defines the first angle of view may be set to a predetermined position in the first angle of view, for example, one vertex in the first angle of view or the center of the first angle of view (i.e., the intersection of the diagonals).

[0036] The process of changing from the first angle of view to the second angle of view includes a rotation process of the affine transformation and a removal process of removing (trimming) a part of the first angle of view. As shown in FIG. 4, the rotation process is a process of converting image information of an image at a first angle of view G1 into image information at a second angle of view G2 obtained by rotating the image from the first angle of view. As shown in FIG. 5, the removal process is a process of converting image information of an image with a first angle of view G1 into image information of an image with a second angle of view G2 that is narrower than the first angle of view. The transformed image shown in FIG. 4, as well as the rotation process and removal process, will be described in detail later.

[0037] When image information is converted, the position of the subject area in the angle of view changes, so when using the converted image data, it is necessary to take into account the change in the position of the subject area. In consideration of this point, the present invention executes a process (more specifically, an output process or an update process described below) for information regarding the position of the subject area in the changed angle of view for the image data whose image information has been converted. This allows the converted image data to be used appropriately and correctly.

[0038] <<Regarding the Processing Apparatus According to the First Embodiment of the Present Invention>> As shown in FIG. 6, a processing device according to a first embodiment of the present invention (hereinafter referred to as processing device 10) constitutes a processing system S together with a user-side device 12 and a learning device 14. The processing system S performs machine learning according to a learning purpose specified by a user or the like, and provides the user with an inference model obtained as a result of the learning. By using the inference model, the user can determine or predict the type and state of a subject in an image.

[0039] The user-side device 12 is configured by, for example, a user-owned PC (Personal Computer), a communication terminal, a camera with a communication function, etc. The user-side device 12 provides image data of a captured image to the processing device 10, and more specifically, transmits the image data to the processing device 10 via the network N. The user-side device 12 provides image data including image information and additional information. The image data may not include any additional information when the processing device 10 acquires the image data from the user-side device 12, and the processing device 10 may add the additional information to the image data after acquiring the image data.

[0040] The processing device 10 is configured by a computer and is communicably connected to the user-side device 12 and the learning device 14. The processing device 10 acquires image data from the user-side device 12 and performs a series of data processing on the image data, including the above-mentioned conversion of image information (specifically, changing the angle of view).

[0041] In addition, the processing device 10 selects image data that meets predetermined conditions based on annotations from a large number of image data including image data before conversion and image data after conversion, and creates training data for machine learning from the selected image data.

[0042] The process of creating teacher data from the selected image data may be executed by a device other than the processing device 10. Furthermore, the annotation (selection) of the image data may be performed before the image information of the image data is converted. In this case, the image information of the image data selected by the annotation may be converted (i.e., the angle of view may be changed) to increase the amount of teacher data, etc.

[0043] The converted image data may be transmitted from the processing device 10 to the user-side device 12. Upon receiving the converted image data, the user-side device 12 displays an image based on the image information included in the image data, i.e., an image of the second angle of view, on the display of the user-side device 12.

[0044] Here, if the second angle of view exceeds the size that can be displayed on the display of the user side device 12, the processing device 10 sets the range of the second angle of view to be displayed on the display of the user side device 12 according to the size of the display. Similarly, if the aspect ratio of the second angle of view does not match the aspect ratio of the display, the processing device 10 sets the range of the second angle of view to be displayed on the display of the user side device 12 according to the aspect ratio of the display. Then, the processing device 10 adds information about the set display range to the additional information of the image data to be transmitted to the user-side device 12.

[0045] The learning device 14 is configured by a computer and performs machine learning using the training data created by the processing device 10. Machine learning is a technology that learns patterns and judgment criteria from data and predicts and judges unknown phenomena based on them, as well as an analytical technology related to artificial intelligence. For example, machine learning is performed to identify the correspondence between the image features indicated by image data, which serves as training data, and the accompanying information contained in the image data. More specifically, the correspondence between the features of the subject in the image and the type of subject indicated by the accompanying information is learned, and an inference model that determines or predicts the type of subject in the image is constructed based on the learning results.

[0046] The inference model constructed by machine learning can be any mathematical model, such as neural networks, convolutional neural networks, recurrent neural networks, attention, transformers, generative adversarial networks, deep learning neural networks, Boltzmann machines, matrix factorization, factorization machines, m-way factorization machines, field-aware factorization machines, field-aware neural factorization machines, support vector machines, Bayesian networks, decision trees, and random forests.

[0047] 6, the processing device 10 and the learning device 14 exist independently as separate devices, but are not limited to this and may be integrated into a single device. Also, the user-side device 12 may be provided with some or all of the functions of the processing device 10.

[0048] To explain an example of the configuration of the processing device 10, the processing device 10 is realized by a processor and a program executable by the processor, and is configured by, for example, a general-purpose computer. As shown in Fig. 6, the computer configuring the processing device 10 includes a processor 10A, a memory 10B, and a communication interface 10C.

[0049] The processor 10A is configured by, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a tensor processing unit (TPU). The memory 10B is configured by, for example, semiconductor memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The communication interface 10C is configured by, for example, a network interface card or a communication interface board.

[0050] A program for executing a series of processes on image data (hereinafter referred to as an image processing program) is installed in the computer constituting the processing device 10. The image processing program is a program for causing the computer to execute the processing method of the present invention. In other words, when the processor 10A reads and executes the image processing program, the computer including the processor 10A functions as the processing device of the present invention.

[0051] The image processing program may be acquired by reading it from a computer-readable recording medium, or may be acquired by receiving (downloading) it via a communication network such as the Internet or an intranet.

[0052] 6, the processing system S is provided with a storage device 16 that stores image data acquired by the processing device 10 from the user device 12. A plurality of image data are stored in the storage device 16 as an image database DB, and the processing device 10 is configured to be able to read the image data stored in the storage device 16. The image data stored in the image database DB also includes image data whose image information has been converted by the processing device 10.

[0053] The image data stored in the storage device 16 may include image data obtained by reading and digitizing a printed (developed) analog photograph using a scanner or the like. The source of the image data is not particularly limited, and may be a device other than the user-side device 12, for example, another device (not shown) present on the network to which the processing device 10 is connected. The storage device 16 may be installed within the processing device 10, or may be provided on the side of a third computer (for example, an external server) that can communicate with the processing device 10.

[0054] 7, the processing device 10 has an acquisition unit 21, a first conversion unit 22, an output unit 23, a first addition unit 24, a second addition unit 25, a second conversion unit 26, an update unit 27, and a storage unit 28. These functional units are realized by cooperation between hardware devices included in the computer constituting the processing device 10 and software including the image processing program described above.

[0055] The acquisition unit 21 acquires image data from the user side device 12 etc. The acquired image data is stored in the storage device 16 and accumulated in the image database DB.

[0056] The first conversion unit 22 reads image data from the storage device 16 and performs a rotation process as a first conversion process on the image data. The amount of rotation from the first angle of view G1 to the second angle of view G2 in the rotation process, i.e., the rotation angle, is set within a rotation range greater than 0° and less than 360°. The second angle of view G2 determined by the rotation process is determined as a range surrounding the first angle of view G1x after rotation, more specifically, as a rectangular range circumscribing the first angle of view G1x after rotation, as shown in FIG.

[0057] The rotation angle in the rotation process can be determined arbitrarily within the above rotation range, and may be set randomly by the processing device 10. Alternatively, the user may specify the rotation angle through the user-side device 12. In FIG. 4, the rotation angle is 45°.

[0058] Furthermore, when the rotation angle satisfies a predetermined condition, specifically, when the rotation angle is within a rotation range excluding 90°, 180°, and 270°, the size of the second angle of view is a range that encompasses the first angle of view G1x. Therefore, the size of the second angle of view in this case is larger than the size of the first angle of view. In this case, as shown in FIG. 4, image P2 of the second angle of view includes a rotated image RP obtained by rotating image P1 of the first angle of view, and complementary images HP adjacent to the four sides of rotated image RP.

[0059] The complementary image HP is a pseudo image that complements the portion of the image P2 of the second angle of view other than the rotated image RP, and is, for example, an image filled with a single or multiple colors, or an image with a predetermined pattern. The color scheme of the complementary image HP is not particularly limited, but as shown in Figure 4, the color of the complementary image HP may be set to the same color as the pixels on the side (edge) of the rotated image RP that the complementary image HP is adjacent to.

[0060] Furthermore, depending on the rotation angle, the aspect ratio of the second angle of view G2 may differ from the aspect ratio of the first angle of view. The image data that has undergone the rotation process is then stored in the storage device 16 and accumulated in the image database DB.

[0061] When the rotation process is performed, the output unit 23 outputs new position information regarding the position of the subject area in the second angle of view G2. The "new position information" is, for example, position information regarding the position of the converted area Ax in the second angle of view G2. Here, the position of the converted area Ax in the second angle of view G2 is expressed in a coordinate space that defines the second angle of view G2.

[0062] 4, the converted area Ax is an area (indicated by a double dashed line in the figure) that is larger than the subject area Ar at the first angle of view G2 (indicated by a dashed line in the figure) and that includes the subject area Ar, more specifically, an area that circumscribes the subject area Ar. The converted area Ax is set by the output unit 23 when the rotation process is executed.

[0063] New position information, i.e., position information on the transformed area Ax, is output based on information on the rotation from the first angle of view to the second angle of view (hereinafter, rotation information) and position information on the position of the subject area A3 at the first angle of view before the rotation process. Here, the position of the subject area Ar at the first angle of view before the rotation process is represented in a coordinate space that defines the first angle of view G1 before the rotation process.

[0064] As an example of a mode for outputting new position information, the position information of the subject area in the incidental information of the image data on which rotation processing has been performed may be rewritten with the position information of the converted area Ax, as shown in Fig. 8. Alternatively, as shown in Fig. 9, the position information of the subject area Ar before rotation processing may be included in the incidental information of the image data on which rotation processing has been performed, together with the position information of the converted area Ax. The image data on which the rotation process has been performed refers to image data that includes image information of the image P2 with the second angle of view, and will be referred to as image data after the rotation process below.

[0065] As described above, the image P2 with the second angle of view can be displayed on the display of the user-side device 12. On the other hand, if the rotation angle in the rotation process satisfies a predetermined condition, the second angle of view G2 becomes larger than the first angle of view G1, and as a result, the second angle of view G2 may exceed the size that can be displayed on the user-side device 12. In this case, the output unit 23 may set a range to be displayed on the user-side device 12 within the second angle of view G2, and output information about the set display range together with position information of the converted area Ax.

[0066] When a rotation process is executed, the first adding unit 24 adds information related to the rotation process to the incidental information included in the image data after the rotation process. The information related to the rotation process is information related to the rotation from the first angle of view to the second angle of view, and specifically, information indicating the rotation angle, the rotation center, and the execution history of the rotation process. The information regarding the rotation process does not need to be the same as the rotation information that the output unit 23 refers to when outputting new position information.

[0067] When the rotation process is performed and the rotation angle satisfies a predetermined condition, the second adding unit 25 adds information about the complementary image HP to the incidental information included in the image data after the rotation process. The information about the complementary image HP includes information about the position of the area of ​​the complementary image HP in the second angle of view G2, and information indicating that the image located in that area is the complementary image HP.

[0068] The second conversion unit 26 reads the image data from the storage device 16 and performs a removal process as a second conversion process on the image data. The degree of removal from the first angle of view G1 to the second angle of view G2 in the removal process is set within a range of more than 0 and less than 100, where the number of pixels in the first angle of view is 100.

[0069] In the removal process, the second angle of view G2 is a rectangular area, and is set so that, for example, part or all of at least one subject (e.g., a main subject) included in the first angle of view G1 remains within the second angle of view G2. In other words, in the removal process, the area of ​​the first angle of view G1 that is outside the second angle of view G2 is removed, as shown in FIG.

[0070] The update unit 27 executes the first update process when a subject area including the target subject at the first angle of view G1 is removed by a predetermined percentage or more at the second angle of view G2 through the removal process. The target subject is a subject included in the first angle of view, whose subject area is removed by a predetermined percentage or more at the second angle of view through the removal process. The first update process is a process of updating incidental information included in the image data on which the removal process has been executed, and specifically, a process of deleting information related to the attributes of the target subject from the incidental information.

[0071] Taking the case shown in Fig. 10 as an example, suppose that the first angle of view G1 of image data before the removal process includes four subjects, and the types (categories) of the subjects are "person," "dog," "tree," and "cloud." Here, if the removal process removes a predetermined percentage or more of the subject areas of "dog," "tree," and "cloud" at the second angle of view G2, the three subjects "dog," "tree," and "cloud" correspond to target subjects. In this case, a first update process is executed, and as a result, the attribute information of each of the three target subjects is deleted as shown in Fig. 10.

[0072] When the first angle of view G1 of the image data before the removal process includes a plurality of subjects, the update unit 27 calculates the removal rate at the second angle of view G1 for each subject region using the following formula (1). Removal ratio at the second angle of view = (k1-k2) / k1 × 100 Equation (1) In formula (1), k1 indicates the number of pixels in each subject region in the first angle of view G1, and k2 indicates the number of pixels in each subject region in the second angle of view G2. For example, if the number of pixels in the subject region of the "dog" in the first angle of view G1 is 2000, and the number of pixels in the subject region of the "dog" in the second angle of view is reduced to 500 through the removal process, the removal ratio is 75 (=1500 / 2000×100).

[0073] Furthermore, the update unit 27 determines whether the calculated removal ratio for each subject region is equal to or greater than a predetermined ratio, and sets the subject included in the subject region for which the removal ratio is equal to or greater than the predetermined ratio as the target subject. Here, the predetermined ratio that serves as the criterion for determining whether or not a subject corresponds to a target subject is set according to the attribute of the target subject (specifically, type, etc.), and may differ for each attribute. In this case, it is possible to appropriately determine whether or not a subject should be set as the target subject (in other words, whether or not a subject should be subject to the first update process) according to the attribute of the subject.

[0074] Furthermore, if the target subject is a person, the above-mentioned predetermined ratio may differ depending on the direction of the person's face. Whether or not the person who is the subject should be the target subject (in other words, whether or not the person should be the target of the first update process) can be appropriately determined depending on the direction of the person's face.

[0075] If the target subject is present, the update unit 27 executes a first update process, in which the attribute information of the target subject is deleted from the incidental information included in the image data after the removal process, as shown in Fig. 10. Here, the image data after the removal process refers to the image data on which the removal process has been executed.

[0076] Note that the removal process may result in a portion of the subject at the first angle of view being extracted at the second angle of view, and for example, in the case shown in Fig. 10, the upper body portion of the subject "male" is extracted by the removal process. When a portion of the subject is extracted by the removal process in this way, the update unit 27 may update the attribute information of the subject from which the portion has been extracted to content corresponding to the extracted portion ("male upper body" in Fig. 10) as shown in Fig. 10.

[0077] When the removal process is executed, the storage unit 28 stores the pre-conversion image data in the storage device 16 and accumulates it in the image database DB. The pre-conversion image data is image data that includes image information of the image P1 of the first angle of view before the removal process (before conversion) and supplementary information on the position (more specifically, the relative position) of the second angle of view G2 within the first angle of view G1.

[0078] It is preferable that the pre-conversion image data be converted into data of a smaller capacity by, for example, lowering the image quality of the image P1 of the first angle of view before being stored in the storage device 16. In this case, the data storage capacity can be reduced, and the pre-conversion image data can be easily read. Furthermore, it is preferable that the pre-conversion image data be stored as part of the image data after the removal process, or be stored in association with the image data after the removal process in the storage device 16. In this case, it is easy to know which image data is the original data of the image data after the removal process.

[0079] <<Image Data Processing Procedure According to the First Embodiment of the Present Invention>> Next, a processing flow of image data using the processing device 10 according to the first embodiment of the present invention will be described. In the processing flow described below, image data is processed using the processing method of the present invention. In other words, each step in the processing flow described below corresponds to a component of the processing method of the present invention. The processing flow described below is merely an example, and unnecessary steps may be deleted, new steps may be added, or the order in which steps are executed may be changed, without departing from the spirit of the present invention.

[0080] The processing flow is started, for example, when the processing device 10 receives an execution request transmitted from the user-side device 12. During the processing flow, the processing device 10 acquires image data from the user-side device 12 or the like, and stores the acquired image data in the storage device 16. An image database DB is constructed in the storage device 16 by accumulating image data.

[0081] The processing flow includes a rotation processing flow shown in Fig. 11 and a removal processing flow shown in Fig. 12. The rotation processing flow is a flow for executing rotation processing when converting image information. The removal processing flow is a flow for executing removal processing when converting image information. Each processing flow will be explained below.

[0082] (Rotation processing flow) Each step (process) in the rotation processing flow is executed by the processor 10A of the processing device 10. The processor 10A reads an image processing program and executes a series of data processes related to the rotation processing flow.

[0083] In the rotation processing flow, processor 10A reads image data from storage device 16 and performs a rotation process on the read image data (S001). This step S001 corresponds to a conversion step in the rotation processing flow. In step S001, image information included in the image data, i.e., image P1 with a first angle of view, is converted into image information of image P2 with a second angle of view rotated from the first angle of view.

[0084] Next, the processor 10A determines whether the rotation angle in the rotation process satisfies a predetermined condition, more specifically, whether the rotation angle is within a rotation range greater than 0° and less than 360°, excluding 90°, 180°, and 270° (S002).

[0085] If the rotation angle satisfies a predetermined condition, the processor 10A executes an output step (S003). In the output step, the processor 10A sets a converted area Ax at the second angle of view G2 (S004), as shown in Fig. 11. The converted area Ax is set based on position information of the subject area at the first angle of view G1 before the rotation process and information (rotation information) related to the rotation from the first angle of view G1 to the second angle of view G2, and is circumscribed with the subject area Ar at the second angle of view G2 (see Fig. 4).

[0086] In the output step, the processor 10A executes a complementation step in which complemented images HP, which are pseudo images, are placed adjacent to each of the four sides of the rotated image RP obtained by rotating the image P1 of the first angle of view (S005). As a result, an image P2 of the second angle of view is generated in which the portion other than the rotated image RP is complemented with the complemented images HP.

[0087] Furthermore, in the output step, processor 10A outputs new position information regarding the position of the subject area at second angle of view G2 based on the rotation information and the position information of the subject area at first angle of view before the rotation process (S006). Specifically, processor 10A outputs the position information of post-conversion area Ax set in step S004 as the new position information.

[0088] When outputting new position information, the processor 10A may rewrite the position information of the subject area included in the incidental information of the image data after the rotation process with the position information of the converted area Ax (see FIG. 8). Furthermore, the position information of the converted area Ax may be associated with the image data after the rotation process and stored in a data file separate from the image data.

[0089] By outputting new position information for the subject area as described above, even if the position of the subject area changes as a result of a change in the angle of view due to the rotation process, the position of the changed subject area (specifically, the transformed area Ax) can be appropriately identified. This makes it possible to accurately grasp the correspondence between the position of the subject area and the attributes of the subject included in the subject area for the image data after the rotation process. As a result, when machine learning is performed using training data created from the image data after the rotation process, highly accurate learning results can be obtained.

[0090] Furthermore, the transformed area Ax is set at the second angle of view G2 to be larger than the subject area Ar at the first angle of view and to include the subject area Ar (see FIG. 4). Since the transformed area Ax includes the same subject as the subject area Ar included in the transformed area Ax, even if the position of the subject area moves due to the rotation process, the subject included in the subject area can be properly detected.

[0091] Returning to the explanation of the rotation processing flow, when the rotation processing is executed, the processor 10A executes a first adding step and a second adding step (S007, S008). In the first adding step, the processor 10A adds information about the rotation processing to the additional information of the image data after the rotation processing.

[0092] By adding information about the rotation process in the first adding step, it is possible to easily identify that the image data to which the information is added is image data after the rotation process. As a result, for example, when machine learning is performed using training data created from image data after the rotation process, it is possible to easily understand that the training data was created from image data after the rotation process. Furthermore, it is possible to identify the correspondence between the rotation angle, rotation center, etc. indicated by the information about the rotation process and the accuracy of machine learning using training data created from image data after the rotation process.

[0093] In the second addition step, the processor 10A adds information about the complementary image HP to the additional information of the image data after the rotation process. By adding information about the complementary image HP, it is possible to easily determine whether the image data is an image that includes the complementary image HP. As a result, when selecting (annotating) image data for creating training data, the presence or absence of the complementary image HP can be taken into consideration when selecting image data.

[0094] Furthermore, when machine learning is performed using training data created from image data that includes a complementary image HP, the presence or absence of a complementary image HP and its location can be identified based on information about the complementary image HP, thereby preventing the complementary image HP from being mistakenly recognized as a correct image during learning.

[0095] When the series of steps S001 to S008 described above is completed, the rotation processing flow ends.

[0096] (Removal process flow) Each step (process) in the removal processing flow is executed by the processor 10 A. The processor 10 A reads an image processing program and executes a series of data processes related to the removal processing flow.

[0097] In the removal process flow, processor 10A reads image data from storage device 16 and performs removal processing on the read image data (S011). This step S011 corresponds to a conversion step in the removal process flow. In step S011, image information included in the image data, i.e., image information of an image with a first angle of view G1, is converted into image information with a second angle of view G2 that is narrower than the first angle of view.

[0098] Next, the processor 10A determines whether or not the object area in the first angle of view G1 has been removed by a predetermined ratio or more in the second angle of view G2 by the removal process (i.e., the conversion process) (S012). If the first angle of view G1 includes multiple object areas, the processor 10A repeats steps S012 and subsequent steps for each object area.

[0099] If a subject area that has been removed by the removal process by a predetermined percentage or more, i.e., a subject area of ​​the target subject, exists, the processor 10A executes an update step (S013). In the update step, the processor 10A updates the incidental information of the image data after the removal process, and specifically, deletes the attribute information of the target subject from the incidental information (see FIG. 10).

[0100] A target subject from which a predetermined percentage or more of the subject area has been removed is difficult to identify in the second angle of view G2, or exists outside the second angle of view G2, so attribute information is unnecessary. Therefore, by performing the update process as described above and deleting the attribute information of the target subject from the incidental information, the incidental information of the image data after the removal process can be updated to appropriate information according to the second angle of view G2. Furthermore, when machine learning is performed using training data created from image data after the removal process, deleting the attribute information of the target subject can reduce the impact of the target subject on the learning results. As a result, highly accurate learning results can be obtained.

[0101] Returning to the explanation of the removal process flow, when the processor 10A executes the removal process (i.e., the conversion step), the processor 10A also executes a storage step (S014). In the storage step, the processor 10A stores the pre-conversion image data in the storage device 16.

[0102] By storing the pre-conversion image data as described above, machine learning can be performed using training data created from the pre-conversion image data. This machine learning can identify the relationship between the first angle of view P1 and the second angle of view P2, or more simply, the tendency regarding the position of the area containing the main subject at the first angle of view P1. In other words, as a result of learning, a model for detecting the main subject in an image can be constructed.

[0103] From the viewpoint of data storage capacity and data read speed, it is preferable that the pre-conversion image data be converted into lower-capacity data before being stored in storage device 16. It is also preferable that the pre-conversion image data be stored in storage device 16 in association with the image data after the removal process. The low-capacity pre-conversion image data may be recorded as a thumbnail image together with the converted image data. In this case, it is more preferable that the additional information of the image data after the removal process includes position information of the subject area in the pre-conversion image data, similar to the image data shown in FIG. 9.

[0104] The removal processing flow ends when the series of steps S011 to S014 described above is completed.

[0105] <<Other embodiments>> The embodiments described above are specific examples given to clearly explain the processing method, processing device, and program of the present invention, and are merely examples, and other embodiments may be considered.

[0106] In the first embodiment described above, in the updating step when the removal process is performed, the attribute information of the target subject from which a predetermined percentage or more of the subject area has been removed is deleted from the incidental information of the image data after the removal process. However, the method of updating the incidental information in the updating step may be a method other than the above-described method.

[0107] A second embodiment of the present invention will be described below, which is a different embodiment from the above-described embodiment of the updating process. The following mainly describes the configuration of the second embodiment that is different from the first embodiment, and omits a description of the configuration that is common to the first embodiment.

[0108] In the updating step according to the second embodiment, the processor 10A executes a third adding step instead of deleting the attribute information of the target subject. The third adding step is a process of adding information about the degree to which the subject area of ​​the target subject remains in the second angle of view (hereinafter, “remaining degree information”) to the additional information of the image data from which the removal process has been executed.

[0109] The remaining degree information is, for example, information indicating a remaining ratio as shown in FIG. 13. The remaining ratio is the ratio of the subject area in the second angle of view to the subject area in the first angle of view for each subject including the target subject. Specifically, when the above-mentioned removal ratio is j, the remaining ratio can be calculated using the following formula (2). For example, when the residual image ratio is 100, this means that the target subject remains completely in the second angle of view. On the other hand, when the residual image ratio is 0, this means that the target subject has been completely removed in the second angle of view. Remaining ratio=100-j (2)

[0110] The remaining degree information is not limited to the remaining ratio, and may be the number of pixels (number of remaining pixels) constituting the subject area in the second angle of view G2, as shown in Fig. 14. In this case, the remaining degree information may be a set of the number of remaining pixels and the number of pixels in the subject area in the first angle of view G1 (i.e., the subject area before the removal process). Alternatively, the remaining degree information may be a set of the number of remaining pixels and the number of pixels removed by the removal process.

[0111] Furthermore, information indicating whether the subject area remains within the second angle of view G2, in other words, whether the subject area is outside the second angle of view G2, may be added as remaining degree information. Furthermore, if the target subject is a person, the proportion of the person's face in the subject area remaining within the second angle of view G2 may be included in the remaining degree information.

[0112] As described above, by adding the remaining degree information regarding the subject area of ​​the target subject to the incidental information of the image data after the removal process, it is possible to easily grasp the degree to which the subject area of ​​the target subject has been removed in the removal process. Furthermore, by referring to the remaining degree information, it is possible to determine whether or not the image data after the removal process is suitable for use.

[0113] Specifically, the image data after the removal process may be used to create training data, provided that the numerical value indicated by the remaining degree information (e.g., the remaining ratio or the number of remaining pixels) is equal to or greater than a threshold value. This is because the higher the numerical value, the easier it is to identify the target subject included in the subject area. In light of this, if training data is created using image data whose numerical value has reached a predetermined standard (threshold value), the reliability (credibility) of the training data will be improved, allowing for accurate machine learning.

[0114] On the other hand, the image data after the removal process may be used to create training data, provided that the remaining ratio, number of remaining pixels, or ratio of human faces in the subject area indicated by the remaining degree information is less than a threshold. This is because, for example, if the subject is a person, using image data that clearly shows that person without permission could result in a problem with portrait rights. In light of this, selecting image data whose values ​​do not reach a predetermined standard (threshold) can avoid problems related to the subject's rights.

[0115] As described above, the remaining degree information is referenced when selecting image data from the image database DB whose remaining degree of the subject region satisfies the set condition (specifically, the condition that the remaining degree is above or below the threshold). By referring to the remaining degree information, it is possible to select image data suitable for use.

[0116] Furthermore, in the second embodiment, the attribute information of a subject from which a predetermined percentage or more of the subject area has been removed by the removal process, i.e., the target subject, is not deleted but remains in the incidental information. Therefore, when performing machine learning using training data created from image data after the removal process, the attribute information of the target subject can be used. This makes it possible to perform machine learning by utilizing attribute information of subjects outside the second angle of view G2 in addition to attribute information of subjects included in the second angle of view G2. As a result, it is possible to perform machine learning by taking into account, for example, the shooting location or scene of the original image indicated by the image data before the removal process, in other words, the background captured in the image P1 of the first angle of view.

[0117] In the above-described embodiment, the image data whose image information has been converted (i.e., the image data on which the rotation process or the removal process has been performed) is used to create training data for machine learning. However, the image data may be used for purposes other than creating training data.

[0118] Furthermore, the processor of the processing device of the present invention may be provided in each of the user side device 12 and a server computer that can communicate with the user side device 12. Furthermore, when there are multiple processors, the multiple processors may work together.

[0119] The processor included in the processing device of the present invention includes various types of processors, including, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various processors also include PLDs (Programmable Logic Devices), which are processors whose circuit configuration can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays). Furthermore, various processors include dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits), which are processors having circuit configurations designed specifically for performing specific processes.

[0120] Furthermore, one functional unit possessed by the processing device of the present invention may be configured by one of the various processors described above, or may be configured by a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Furthermore, the multiple functional units of the processing device of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be combined into one processor. Furthermore, as in the above-described embodiment, one processor may be configured by combining one or more CPUs and software, and this processor may function as multiple functional units.

[0121] Furthermore, a processor may be used that realizes the functions of the entire system including multiple functional units in the processing device of the present invention on a single IC (Integrated Circuit) chip, as typified by an SoC (System on Chip), etc. Furthermore, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0122] 10 Processing equipment 10A processor 10B memory 10C communication interface 12 User side equipment 14 Learning Device 16 Storage device 21 Acquisition Department 22 First conversion unit 23 Output section 24 First Additional Section 25 Second Additional Section 26 Second conversion unit 27 Update section 28 Memory section Ar subject area Ax transformed area G1 1st angle of view First angle of view after G1x rotation G2 2nd angle of view DB Image Database HP complementary image N Network P1 First angle of view image P2 Second angle of view image RP Rotated Image S Processing System

Claims

1. A processing method for processing image data including image information of an image having a first angle of view including a subject and additional information added to the image, a conversion step of converting image information of the image at the first angle of view into image information of an image at a second angle of view obtained by rotating the image at the first angle of view; an output step of outputting, when the conversion step is executed, new position information regarding a position of the area of ​​the subject in the second angle of view, based on information regarding the rotation from the first angle of view to the second angle of view and position information regarding a position of the area of ​​the subject in the first angle of view, a first adding step of adding information related to a rotation process from the first angle of view to the second angle of view to the incidental information of the image data including image information of the image with the second angle of view when the converting step is executed, The processing method, wherein the information regarding the rotation process from the first angle of view to the second angle of view is information indicating a rotation angle and a rotation center in the rotation process, and an execution history of the rotation process.

2. The processing method according to claim 1 , wherein the output step is executed when a rotation angle from the first angle of view to the second angle of view satisfies a predetermined condition.

3. The processing method according to claim 2, wherein the output step sets a transformed area at the second angle of view that is larger than the area of ​​the subject at the first angle of view and includes the area of ​​the subject, and outputs position information regarding the transformed area at the second angle of view as the new position information.

4. The processing method according to claim 2 , wherein the predetermined condition is that the rotation angle is within a rotation range of more than 0° and less than 360°, excluding 90°, 180°, and 270°.

5. the converting step includes a complementing step of complementing, with a complement image, a portion of the image of the second angle of view other than a rotated image obtained by rotating the image of the first angle of view, The processing method according to claim 1 , further comprising a second adding step of adding information about the complementary image to the supplementary information of the image data including the image information of the image of the second angle of view when the converting step is executed.

6. The processing method according to claim 1 , wherein the image data is used to create training data used in machine learning.

7. A processing device that processes image data including image information of an image having a first angle of view including a subject and additional information added to the image, a conversion step of converting image information of the image at the first angle of view into image information of an image at a second angle of view obtained by rotating the image at the first angle of view; an output step of outputting new position information regarding a position of the area of ​​the subject in the second angle of view, based on information regarding the rotation from the first angle of view to the second angle of view and position information regarding a position of the area of ​​the subject in the first angle of view, when the conversion step has been executed; configured to run when the conversion step is executed, a first addition step is further executed of adding information related to a rotation process from the first angle of view to the second angle of view to the incidental information of the image data including image information of the image with the second angle of view, The processing device, wherein the information related to the rotation process from the first angle of view to the second angle of view is information indicating a rotation angle and a rotation center in the rotation process, and an execution history of the rotation process.

8. A program for causing a computer to execute the processing method according to claim 1.

9. A processing method for processing image data including image information of an image having a first angle of view including a subject and additional information added to the image, a conversion step of converting the image information of the image having the first angle of view into image information of an image having a second angle of view narrower than the first angle of view; an updating step of updating the supplementary information when a subject area including a target subject, which is the subject at the first angle of view, is removed by a predetermined percentage or more at the second angle of view by the conversion step.

10. 10. The processing method according to claim 9, wherein the updating step includes a process of deleting information relating to attributes of the target subject from the auxiliary information, or a process of adding information relating to a degree of remaining of the subject area in the second angle of view to the auxiliary information.

11. 11. The processing method according to claim 10, wherein the information regarding the remaining degree of the subject area at the second angle of view includes a remaining ratio of the subject area at the second angle of view to the subject area at the first angle of view, or a number of pixels constituting the subject area at the second angle of view.

12. The processing method according to claim 11, wherein the information regarding the remaining degree of the subject area at the second angle of view is information that is referenced when selecting image data whose remaining degree satisfies a set condition from an image database in which a plurality of image data are accumulated.

13. The processing method according to claim 9 , wherein the predetermined ratio is set according to an attribute of the target subject.

14. The processing method according to claim 9 , wherein when the target subject is a person, the predetermined ratio varies depending on a face orientation of the person.

15. The processing method according to claim 9 , wherein the subject area is an area specified by coordinates of a plurality of points in the subject area, or an area specified by coordinates of a base point in the subject area and a distance from the base point.

16. 10. The processing method according to claim 9, further comprising a storage step of, when the conversion step is executed, storing pre-conversion image data including the image information of the image of the first angle of view before conversion and the auxiliary information regarding the position of the second angle of view in the first angle of view.

17. A processing device that processes image data including image information of an image having a first angle of view including a subject and additional information added to the image, a conversion step of converting the image information of the image having the first angle of view into image information of an image having a second angle of view narrower than the first angle of view; an updating step of updating the auxiliary information when a subject area including a target subject that is the subject at the first angle of view is removed by a predetermined ratio or more at the second angle of view by the converting step; a processing unit configured to execute

18. A program for causing a computer to execute the processing method according to claim 9.

Images