Image processing method, image processing device, and image processing program

Abstract

The image processing apparatus acquires an image configured by an omni-directional image associated with a positive solution label of an object, executes object sensing processing for sensing the object from the acquired image, calculates the sensing accuracy of the object in the object sensing processing on the basis of the positive solution label, and processes the image when the sensing accuracy is lower than a threshold value. Therefore, the deformation of the object contained in the image is increased.

Description

Technical Field

[0001] The present disclosure relates to a technique for processing an image. Background Art

[0002] Patent document 1 discloses the following technology: an area where an object is more likely to exist is selected from a camera image captured by an omnidirectional camera as a candidate area, the orientation of the candidate area is rotated so that the object contained in the selected candidate area is oriented in a vertical direction, and object sensing processing is performed on the rotated candidate area.

[0003] However, Patent Document 1 is a technique for rotating the orientation of a candidate region so as to reduce the deformation of an object contained in the candidate region, and is not a technique for intentionally increasing the deformation of an object on an image. Therefore, Patent Document 1 cannot generate a learning image for accurately sensing an object from an image with deformation.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: International Publication No. 2013 / 001941 Summary of the invention

[0007] The present disclosure aims to solve such a problem, and an object of the present disclosure is to provide a technology for generating a learning image capable of accurately sensing an object from an image having deformation.

[0008] An image processing method in one embodiment of the present invention is an image processing method in a computer, which acquires an image composed of omnidirectional images, performs object sensing processing of sensing an object from the acquired image, calculates the sensing accuracy of the object in the object sensing processing, processes the image based on the sensing accuracy so that the deformation of the object contained in the image becomes larger, and outputs a processed image that has undergone the processing.

[0009] According to this configuration, it is possible to generate a learning image that can accurately detect an object from an image having a deformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing an example of the structure of the image processing device in the embodiment of the present disclosure.

[0011] Figure 2 2 is a diagram showing a situation in which a viewpoint conversion process is performed.

[0012] Figure 3 This is an explanatory diagram of the viewpoint conversion process.

[0013] Figure 4FIG. 1 is a diagram showing a display screen of a user interface to which an image to which object sensing processing has been performed using a learning model is applied.

[0014] Figure 5 This is a flowchart showing an example of processing performed by the image processing device in the first embodiment.

[0015] Figure 6 1 is a flowchart showing an example of processing in the learning phase in the image processing device 1 .

[0016] Figure 7 This is a diagram showing an image to which the viewpoint conversion process in the second embodiment is applied.

[0017] Figure 8 This is a flowchart showing an example of processing by the image processing device in the second embodiment.

[0018] Fig. 9 FIG. 1 is a diagram showing an example of an object having an easily deformable shape.

[0019] Fig.10 This is a flowchart showing an example of processing by the image processing device in the third embodiment.

[0020] Fig.11 This is a flowchart showing an example of processing by the image processing device in the fourth embodiment.

[0021] Fig.12 This is a block diagram showing an example of the structure of an image processing device in Embodiment 5.

[0022] Fig.13 This is a flowchart showing an example of processing by the image processing device 1A in the fifth embodiment. DETAILED DESCRIPTION

[0023] (Process of realizing one mode of the present disclosure)

[0024] Problems at the construction site include communication problems where specific instructions are not conveyed to workers and explanation of the instructions takes time, and problems with confirming the construction site where manpower is required to inspect the entire construction site and movement to the construction site takes time.

[0025] In order to solve such a problem, it is also considered to install multiple cameras at the construction site, and the remote site supervisor gives instructions to the workers while referring to the images obtained from the multiple cameras. However, at the construction site, as the construction progresses, there will be operations such as removing the installed sensors or installing the removed sensors in other places. Since such operations are time-consuming, it is not practical to install sensors at the construction site. For this reason, the inventors have studied a technology that can confirm the conditions of the construction site in detail from a remote location without installing sensors.

[0026] It is then recognized that when an operation is input to select a certain location in the design drawing of the construction site displayed on the display, if an all-round image of the construction site taken in advance at that location is displayed and a user interface is provided that can set an annotation area for the user to annotate the all-round image, the conditions of the construction site can be confirmed in detail remotely.

[0027] When setting the annotation area, the following method is considered: an omnidirectional image that has been pre-sensed using a learning model is displayed on a display, and when a user inputs an operation of selecting an object on the display, a boundary box set in the object is displayed as the annotation area. In this case, even if the user does not input an operation of displaying a default frame in the omnidirectional image, positioning the frame to the target object, and deforming the frame to match the object, the annotation area can be set, thereby reducing the user's labor.

[0028] Here, in order to generate a learning model that can accurately sense an object from an image with deformation, it is preferable to use such an image as an image for learning.

[0029] However, the conventional technology represented by Patent Document 1 processes the image so that the deformation of the object is alleviated in order to improve the sensing accuracy of the object, and therefore does not provide a technical concept of processing the image so as to intentionally increase the deformation of the object.

[0030] To this end, the inventors obtained the understanding that if an image in which the deformation of the object is expressed more greatly is generated as a learning image, and the learning model is made to learn the image, a learning model capable of sensing the object with high accuracy from such an image can be generated, thereby conceiving of the various methods of the present disclosure.

[0031] (1) An image processing method in one aspect of the present disclosure is an image processing method in a computer, in which an image composed of omnidirectional images is acquired, object sensing processing of sensing an object from the acquired image is performed, the sensing accuracy of the object in the object sensing processing is calculated, and based on the sensing accuracy, the image is processed so that the deformation of the object included in the image becomes larger, and a processed image that has undergone the processing is output.

[0032] According to this structure, the sensing accuracy when the object sensing process is performed on the image is calculated, and based on the calculated sensing accuracy, the image is processed so that the deformation of the object becomes larger. In this way, it is possible to generate a learning image for generating a learning model that can sense the object with high accuracy from an image with large deformation of the object.

[0033] (2) In the image processing method described in (1) above, the image may be an image composed of an omnidirectional image corresponding to a correct answer label of the object, the sensing accuracy may be calculated based on the correct answer label, and the processing may be performed when the sensing accuracy is lower than a threshold value.

[0034] According to this structure, since an image in which object sensing is difficult based on object sensing processing is processed, it is possible to provide an image more suitable for learning a learning model that can accurately sense an object from an image in which the object is greatly deformed. In addition, since the sensing accuracy is calculated based on the correct answer label, it is easy to calculate the sensing accuracy.

[0035] (3) In the image processing method described in (1) or (2) above, the image may include a first image and a second image different from the first image, and the sensing accuracy is a sensing accuracy of a sensing result when the first image is input into a learning model that has been pre-learned in order to perform the object sensing processing, and the image processing is performed on the second image.

[0036] According to this configuration, it is possible to generate a learning image that can improve the sensing accuracy of a learning model that has low sensing accuracy for an object in an omnidirectional image. Therefore, it is possible to efficiently perform learning of the learning model.

[0037] (4) In the image processing method described in (3) above, the processed image may be used to cause the learning model to perform learning.

[0038] According to this configuration, since the learning model is learned using an image in which the object is greatly deformed, it is possible to generate a learning model that can sense the object with high accuracy from the image having deformation.

[0039] (5) In the image processing method described in any one of (2) to (4) above, the sensing accuracy may be calculated for each object category, and the second image may be an image including an object determined in the first image to have the sensing accuracy equal to or less than a threshold value.

[0040] According to this configuration, since an image including an object that is difficult for the learning model to sense is generated as a learning image, the learning model can be made to learn so that the sensing accuracy of the object is improved.

[0041] (6) In the image processing method described in any one of (1) to (5) above, the processing of the image may include changing a default viewpoint of the image to a randomly set viewpoint.

[0042] According to this configuration, since the image is processed by randomly changing the viewpoint, the object displayed at a position with a small deformation before processing is more likely to be displayed at a position with a large deformation, and an image with a larger deformation of the object can be generated.

[0043] (7) In the image processing method described in any one of (1) to (5) above, the processing of the image may also include: determining an interval between two binding frames with the longest distance among a plurality of correct answer labels set in the image, and setting the viewpoint of the image at the midpoint of the interval.

[0044] According to this configuration, since the image is processed so that the object to which the correct answer label is assigned is displayed at the end of the image, it is possible to generate an image in which the deformation of the object is shown more greatly.

[0045] (8) In the image processing method described in any one of (1) to (7) above, the processing of the image may also include: determining whether the image contains an object whose aspect ratio and at least one of its size exceed a reference value; and in a case where it is determined that the image contains an object exceeding the reference value, increasing the number of processed images compared to a case where it is determined that the image does not contain the object.

[0046] In an omnidirectional image, there is a high probability that vertically long objects, horizontally long objects, and large objects are displayed in a deformed manner. According to this structure, when it is determined that such an object is included in the image, more processed images are generated compared to when it is not included, so it is possible to efficiently generate a learning image that can improve the sensing accuracy of such an object.

[0047] (9) In the image processing method described in (1) or (8) above, the object detection process may be a rule-based object detection process, and the image processing may include performing the image processing on the image subjected to the object detection process.

[0048] According to this configuration, since an image determined as an object with low sensing accuracy by the rule-based object sensing process is processed, it is possible to generate a learning image including an object that is difficult to sense.

[0049] (10) In the image processing method described in any one of (1) to (9) above, the processing of the image may also include processing the image by performing a viewpoint conversion process so that the deformation of the object becomes larger, and the viewpoint conversion process includes: projecting the image onto a unit sphere; setting a new viewpoint based on the projected image; and unfolding the projected image along a plane so that the new viewpoint becomes the center.

[0050] According to this configuration, since the image is projected onto the unit sphere, a new viewpoint is set in the projected image, and thus setting of the new viewpoint becomes easy.

[0051] (11) In another embodiment of the present invention, an image processing device includes a processor, which performs the following processing: acquiring an image composed of omnidirectional images, performing object sensing processing of sensing an object from the acquired image, calculating the sensing accuracy of the object in the object sensing processing, processing the image based on the sensing accuracy so that the deformation of the object included in the image becomes larger, and outputting a processed image that has undergone the processing.

[0052] According to this configuration, it is possible to provide an image processing device capable of generating a learning image for accurately sensing an object from an image having a large deformation.

[0053] (12) An image processing program according to another aspect of the present disclosure causes a computer to execute the image processing method described in any one of (1) to (10) above.

[0054] According to this configuration, it is possible to provide an image processing program capable of generating a learning image for accurately sensing an object from an image having a large deformation.

[0055] The present disclosure can also be realized as an information processing system that operates by such an information processing program. In addition, such a computer program can of course be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0056] In addition, the embodiments described below all represent a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, the order of steps, etc. shown in the following embodiments are examples and are not intended to limit the present disclosure. In addition, among the structural elements in the following embodiments, the structural elements that are not recorded in the independent claims representing the highest concept are described as arbitrary structural elements. In addition, in all embodiments, the various contents can also be combined.

[0057] (Implementation method 1)

[0058] Figure 1 1 is a block diagram showing an example of the structure of the image processing device 1 in the embodiment of the present disclosure. The image processing device 1 is a computer including a processor 10 and a memory 20. The processor 10 is, for example, a central processing unit (CPU). The processor 10 includes: an acquisition unit 11, a sensing unit 12, a verification unit 13, a processing unit 14, an output unit 15, and a learning unit 16. The acquisition unit 11 to the learning unit 16 are implemented by the processor 10 executing an image processing program. The memory 20 is configured by a non-volatile rewritable storage device such as a solid state drive (SSD). The memory 20 includes a verification image database 21, a learning model 22, and a learning image database 23. In addition, in Figure 1 In the example of , all modules are concentrated in one computer, but they can also be dispersed and configured in multiple computers. In this case, multiple computers are connected to each other through the Internet or a local area network so that they can communicate with each other. For example, the learning unit 16 can also be installed in a device different from the image processing device 1, and the memory 20 can also be installed in a device different from the image processing device 1.

[0059] The acquisition unit 11 acquires a verification image from a verification image database 21. The verification image is an image used to verify the sensing accuracy of the object of the learning model 22. Here, an omnidirectional image is used as the verification image. The verification image is associated with the correct label of the object. The verification image is an example of the first image. The correct label includes: a bounding box indicating the position of the object in the verification image, and a classification label indicating the classification to which the object belongs. The omnidirectional image is an image captured by an omnidirectional camera. A conventional camera can only capture the inside of a certain angle, and an omnidirectional camera can capture a 360-degree azimuth, that is, a omnidirectional image of up, down, left, right, front, and back. Since the omnidirectional image is an image obtained by unfolding the image captured by the omnidirectional camera using an unfolding method such as an equidistant cylindrical projection method, it becomes an image whose deformation varies depending on the position. Therefore, when a learning model that has been learned only using images captured by a conventional camera is used to sense an object from an omnidirectional image, the possibility of a decrease in the sensing accuracy of the object increases.

[0060] The sensing unit 12 performs an object sensing process of sensing an object from the verification image acquired by the acquisition unit 11. In detail, the sensing unit 12 performs the object sensing process by inputting the verification image to the learning model 22 to obtain a sensing result. The learning model 22 is a model that has been machine-learned in advance in order to perform the object sensing process. The learning model 22 may be any model as long as it is a model that senses an object from an image, such as a deep neural network or a convolutional neural network. The learning model 22 is generated by performing machine learning on a data set of learning images to which correct labels of objects are assigned.

[0061] The verification unit 13 calculates the sensing accuracy of the object in the learning model 22 based on the correct answer label corresponding to the verification image, and determines whether the calculated sensing accuracy is lower than a threshold value. For example, the sensing accuracy is defined by a correct answer rate, which is a ratio of the total number of objects included in the verification image used for verification as the denominator and the number of objects successfully sensed as the numerator.

[0062] If the classification label of the object output from the learning model 22 as a sensing result is consistent with the classification label included in the correct answer label, the verification unit 13 can determine that the object has been successfully sensed. Alternatively, the verification unit 13 can also determine that the object has been successfully sensed when the classification label of the object output from the learning model 22 as a sensing result is consistent with the classification label included in the correct answer label and the reliability of the object exceeds the reference reliability.

[0063] In addition, when the sensing result output from the learning model 22 includes the reliability of each object classification, the verification unit 13 can determine whether the reliability exceeds the reference reliability by classification. And when the reliability exceeds the reference reliability in all classifications, the verification unit 13 can determine that the object sensing is successful. The so-called classification refers to the type of object.

[0064] The processing unit 14 processes the image based on the sensing accuracy so that the deformation of the object included in the image becomes larger. In detail, when the sensing accuracy calculated by the verification unit 13 is lower than the threshold value, the processing unit 14 processes the learning image so that the deformation of the object included in the verification image becomes larger. In more detail, the processing unit 14 processes the learning image by acquiring the learning image from the learning image database 23 and performing a viewpoint conversion process to change the default viewpoint of the acquired learning image to a randomly set viewpoint. The learning image is an omnidirectional image like the verification image, and a correspondence is established with the correct answer label of the object in advance. The learning image is an example of the second image. Therefore, the correct answer label is also inherited in the processed image obtained by processing the learning image. Thus, the processed image obtained by applying the viewpoint conversion process to the original learning image is generated as the learning image. The so-called default viewpoint refers to a viewpoint set as an initial value, for example, a point parallel to the horizontal plane of the omnidirectional camera and corresponding to the north direction. The viewpoint is located at the center of the omnidirectional image.

[0065] The output unit 15 stores the processed image processed by the processing unit 14 in the learning image database 23 .

[0066] The learning unit 16 uses the processed images stored in the learning image database 23 to make the learning model 22 learn. The learning unit 16 calculates the learning error based on the correct answer label given in the processed image and the reliability output from the learning model 22, and updates the parameters of the learning model 22 so that the learning error is minimized. As a parameter updating method, the error back propagation method can be used. The parameters include weight values ​​and bias values.

[0067] The verification image database 21 stores verification images. The learning model 22 is a learning model 22 to be verified. The learning image database 23 stores learning images.

[0068] Figure 2: is a diagram showing a situation where a viewpoint conversion process is performed. Image G10 is an omnidirectional image before the viewpoint conversion process, and image G20 is an omnidirectional image after the viewpoint conversion process. In this example, in image G20, viewpoint A1 is changed by 180 degrees in the horizontal direction relative to image G10. In images G10 and G20, viewpoint A1 is located at the center of the image. Since images G10 and G20 are omnidirectional images, the deformation varies depending on the position. For example, it can be seen that the deformation of the left and right ends and the upper and lower ends is larger than that of the central part. In image G10 and image G20, object F1 is the same object. It can be seen that object F1 located in the center of the horizontal direction in image G10 moves to the end of the image in image G20, and the deformation increases. In this way, by applying the viewpoint conversion process, the object located in the center of the current image becomes located at the end of the image, and the deformation of the object increases.

[0069] Figure 3 This is an explanatory diagram of the viewpoint conversion process. Image G30 is an omnidirectional image and is represented by the coordinate system of the equidistant cylindrical projection method. The coordinate system of the equidistant cylindrical projection method (an example of a plane) is a two-dimensional coordinate system represented by the u axis in the horizontal direction and the v axis in the vertical direction. The horizontal dimension of image G30 is 2h, and the vertical dimension is h.

[0070] First, the processing unit 14 converts the point Q in the image G30 into a polar coordinate system with a radius of 1. In this case, the point Q (u, v) is expressed by equation (1).

[0071] θ=πu / h, φ=πv / h…(1)

[0072] θ is the zenith angle and φ is the declination angle.

[0073] Next, the processing unit 14 projects the point Q from the polar coordinate system to the three-dimensional orthogonal coordinate system. In this case, the point Q (x, y, z) is expressed by equation (2).

[0074] x=sinθ·cosφ, y=sinθ·sinφ, z=cosφ…(2)

[0075] Next, the processing unit 14 sets the rotation matrices Y(ψy), P(θp), and R(φr) for the three axes of yaw, pitch, and roll. ψy is the rotation angle around the yaw axis, θp is the rotation angle around the pitch axis, and φr is the rotation angle around the roll axis. As a result, as shown in equation (3), point Q(x, y, z) is projected to point Q´(x´, y´, z´).

[0076] [Mathematical formula 1]

[0077]

[0078] Next, the processing unit 14 converts the point Q' from the orthogonal coordinate system to the polar coordinate system using equation (4). θ' is the zenith angle after the conversion, and φ' is the declination angle after the conversion.

[0079] [Mathematical formula 2]

[0080]

[0081] Next, the processing unit 14 converts the point Q' from the polar coordinate system to the coordinate system of the equidistant cylindrical projection method. In this case, the point Q' is expressed by equation (5).

[0082] u´=θ´h / π, v´=φ´h / π…(5)

[0083] The above processing is performed on all points of the image G30 to perform viewpoint conversion processing on the image G30. u' is the coordinate value of the u axis after the viewpoint conversion, and v' is the coordinate value of the v axis after the viewpoint conversion.

[0084] In the first embodiment, the processing unit 14 randomly converts the viewpoint of the image G30 by randomly setting the above-mentioned rotation angles φr, θp, and ψy. Specifically, the processing unit 14 sets the center of the image G30 in the positive cylindrical coordinate system obtained by rotating at the rotation angles φr, θp, and ψy as the viewpoint. In addition, in the embodiments described below, the processing unit 14 performs the viewpoint conversion processing using a method that is not random but corresponding to the embodiment.

[0085] Next, an application example of an image in which object sensing processing has been performed using the learning model 22 will be described. Figure 4 1 is a diagram showing a display screen G1 of a user interface to which an image subjected to object sensing processing by the learning model 22 is applied.

[0086] The display screen G1 is a basic screen of the application for a remote user to check the status of the work site. The display screen G1 includes an image display column R1, an annotation information display column R2, and a design drawing display column R3. In the design drawing display column R3, a design drawing of the work site is displayed, and in the design drawing, a selection icon 201, a shooting location icon 202, and a track 203 are superimposed and displayed. At the work site, a shooting operation using an omnidirectional camera was previously performed by a worker, and the shooting location icon 202 indicates the shooting location of the image captured in the shooting operation. The track 203 indicates the movement track of the worker during the shooting operation.

[0087] The user inputs an operation of selecting a shooting location icon 202 by dragging and dropping the selection icon 201 on the design drawing. Then, an all-round image of the work site photographed at the shooting location indicated by the selected shooting location icon 202 is displayed in the image display column R1. The user sets an annotation area D1 for the image displayed in the image display column R1, and inputs an annotation message for the annotation area D1 into the annotation information display column R2. Thus, the annotation area D1 and the annotation message are shared by each other. As a result, a remote user can confirm in detail the new conditions and precautions of the work site even without moving to the work site.

[0088] The omnidirectional image displayed in the image display field R1 has been previously subjected to object sensing processing by the learning model 22. Therefore, when the user inputs an operation to select an object to be annotated in the omnidirectional image, a bounding box of the object is displayed, and the annotation area D1 can be set based on the bounding box. Thus, the user can set the annotation area D1 even if he does not input an operation to display a frame for setting the annotation area D1 in the image display field R1, move the frame to the position of the target object, and deform the frame to match the shape of the object.

[0089] Figure 5 This is a flowchart showing an example of processing of the image processing device 1 in Embodiment 1. First, in step S1 , the acquisition unit 11 acquires a data set of verification images including a predetermined number of verification images from the verification image database 21 .

[0090] Next, in step S2 , the sensing unit 12 sequentially inputs each of the verification images constituting the data set of the verification images into the learning model 22 , thereby sensing an object included in the verification image.

[0091] Next, in step S3, the verification unit 13 calculates the above-mentioned correct answer rate for the data set of the sensing image acquired in step S1 by comparing the sensing results of the object in the learning model 22 with the correct answer label, and calculates the calculated correct answer rate as the sensing accuracy of the learning model 22.

[0092] Next, the verification unit 13 determines whether the sensing accuracy calculated in step S3 is below a threshold value (step S4). If it is determined that the sensing accuracy is below the threshold value (yes in step S4), the processing unit 14 obtains a data set of learning images including a given number of learning images from the learning image database 23 (step S5).

[0093] Next, the processing unit 14 randomly sets a viewpoint for each learning image (step S6 ). Specifically, as described above, the viewpoint is randomly set by randomly setting the rotation angles φr, θp, and ψy.

[0094] Next, the processing unit 14 generates a processed image in which the default viewpoint is changed to the set viewpoint by performing viewpoint conversion processing on each learning image (step S7). The generated processed image is stored in the learning image database 23. Here, the processing unit 14 may also generate K processed images by randomly setting K (K is an integer greater than or equal to 2) viewpoints for one learning image. Thus, a plurality of processed images representing an object with various deformations are generated from one learning image. As a result, a processed image suitable for learning the learning model 22 can be efficiently generated.

[0095] Next, machine learning in the image processing device 1 will be described. Figure 6 1 is a flowchart showing an example of processing in the learning phase in the image processing device 1 .

[0096] First, in step S21 , the learning unit 16 acquires a data set of processed images including a predetermined number of processed images from the learning image database 23 .

[0097] Next, in step S22 , the learning unit 16 sequentially inputs the dataset of processed images into the learning model 22 to cause the learning model 22 to perform learning.

[0098] Next, in step S23, the learning unit 16 calculates the correct answer rate of object sensing by comparing the sensing results of the object in the learning model 22 with the correct answer labels assigned in the processed images for all processed images acquired in step S22, and calculates the calculated correct answer rate as the sensing accuracy of the learning model 22. The sensing accuracy is calculated in the learning unit 16 in the same way as in the verification unit 13. That is, the learning unit 16 calculates the sensing accuracy as the ratio of the total number of learning images in the data set acquired in step S5 as the denominator and the number of learning images in which the object is successfully sensed as the numerator.

[0099] Next, in step S24, the learning unit 16 determines whether the sensing accuracy is greater than a threshold value. As the threshold value, an appropriate value such as 0.8 or 0.9 can be used. When the sensing accuracy is greater than the threshold value (yes in step S24), the process ends. On the other hand, when the sensing accuracy is less than the threshold value (no in step S24), the process returns to step S21. In this case, the learning unit 16 obtains the data set of processed images from the learning image database 23 again and performs learning of the learning model 22. Here, the data set of processed images used may include or not include the same processed images as the processed images used for learning in the previous cycle.

[0100] Thus, since learning using the processed image is performed in the learning model 22 until the sensing accuracy reaches or exceeds the threshold, it is possible to generate the learning model 22 capable of sensing an object with high accuracy from an omnidirectional image.

[0101] Thus, according to the present embodiment, the sensing accuracy of the learning model 22 sensing the object from the verification image is calculated based on the correct answer label, and when the calculated sensing accuracy is below the threshold, the learning image is processed so that the deformation of the object becomes larger. Thus, it is possible to generate a learning image for generating a learning model capable of sensing the object with high accuracy from an omnidirectional image.

[0102] In the present embodiment, since the image is processed based on the randomly set viewpoint, the object displayed at a position with less deformation before processing is more likely to be displayed at a position with greater deformation, and an image with greater deformation of the object can be generated.

[0103] (Implementation method 2)

[0104] In the second embodiment, the midpoint of the longest interval of two correct answer labels among the intervals of multiple correct answer labels is set as the viewpoint. In addition, in the second embodiment, the same symbols are given to the same components as those in the first embodiment, and the description is omitted. In addition, the block diagram in the second embodiment uses Figure 1 To explain.

[0105] Figure 1 The processing unit 14 shown uses the above-mentioned formula (1) and formula (2) to convert the omnidirectional image before the viewpoint conversion process (hereinafter referred to as the original image) and the bounding box corresponding to the original image onto the unit sphere. The conversion onto the unit sphere is performed using the above-mentioned formula (1) and formula (2). Next, the processing unit 14 determines the two bounding boxes with the longest intervals among the multiple bounding boxes drawn on the unit sphere. Here, Figure 3 As shown, two points representing the positions of two bounding boxes on the unit sphere are set to P and Q. As the position of the bounding box, the center of gravity of the bounding box can be used. The so-called interval refers to the longer arc of the two arcs divided by points P and Q in the great circle 301 passing through points P and Q. Next, the processing unit 14 determines the two bounding boxes with the longest intervals, and sets the midpoint of the intervals of the two bounding boxes as the viewpoint. Next, the processing unit 14 unfolds the original image on the unit sphere so that the viewpoint is located at the center of the positive cylindrical coordinate system. As a result, the object corresponding to the two bounding boxes with the longest intervals is located at the end with greater deformation in the omnidirectional image, and as a result, an omnidirectional image with greater deformation of the object can be obtained.

[0106] Figure 7: is a diagram showing an image G40 to which the viewpoint conversion process in Implementation Example 2 is applied. In image G40, classification labels such as windows, chairs, bathtubs, lamps, mirrors, and doors are associated with bounding boxes. In this example, in the original image, the interval L between the position B1 of the bounding box E1 of the chair and the position B2 of the bounding box E2 of the door is determined to be the longest. Therefore, the original image is expanded so that the midpoint M1 of the interval L becomes the viewpoint, and image G40 is obtained. As a result, the chair and the door are displayed at the two ends of the image G40 with a large deformation, and the deformation of the object is increased.

[0107] Figure 8 is a flowchart showing an example of the processing of the image processing device 1 in the second embodiment. The processing of steps S31 to S35 is similar to that of Figure 5 The processing of steps S1 to S5 is the same as that of steps S36. In step S36, the processing unit 14 specifies the longest section from a plurality of sections of any two bounding boxes among a plurality of bounding boxes associated with the learning image.

[0108] Next, in step S37 , the processing unit 14 sets the midpoint of the interval as the viewpoint.

[0109] Next, in step S38, the learning image is expanded so that the set viewpoint is located at the center, thereby generating a processed image. In this way, a processed image in which the deformation of the object is larger than that before the viewpoint conversion process can be obtained.

[0110] As described above, according to the second embodiment, since the learning image is processed so that the object to which the correct answer label is assigned is displayed at a position with a greater deformation, it is possible to generate a processed image in which the deformation of the object is shown more greatly.

[0111] (Implementation method 3)

[0112] In the third embodiment, processed images are generated so that more processed images are generated that include objects having easily deformable shapes in the omnidirectional image. In the third embodiment, the same reference numerals are given to the same components as those in the first and second embodiments, and the description thereof is omitted. Figure 1 To explain.

[0113] Fig. 9: is a diagram showing an example of an object with an easily deformable shape. An object with an easily deformable shape corresponds to an object whose aspect ratio exceeds a reference aspect ratio and an object whose size exceeds a reference size. The objects shown in images G91 and G92 are objects made of building materials whose aspect ratio exceeds the reference aspect ratio. Image G93 is an object made of building materials whose size exceeds the reference size. An example of an object with an easily deformable shape is a horizontally long sofa, bathtub, ceiling light, door leaf, etc. The aspect ratio includes the ratio of the horizontal side to the vertical side and the ratio of the vertical side to the horizontal side of the bounding box assigned to the object.

[0114] Fig.10 is a flowchart showing an example of processing of the image processing device 1 in the third embodiment. Steps S41 to S44 are Figure 5 Since S1 to S4 are the same, the description thereof will be omitted. In step S45 , the processing unit 14 acquires a learning image from the learning image database 23 .

[0115] Next, in step S46, the processing unit 14 calculates the size and aspect ratio of the object included in the learning image. For example, the processing unit 14 calculates the size of the object based on the area of ​​the bounding box corresponding to the learning image. The processing unit 14 calculates the aspect ratio based on the lengths of the vertical and horizontal sides of the bounding box corresponding to the learning image.

[0116] Next, in step S47, the processing unit 14 determines whether the learning image contains an object whose size exceeds the reference size or whose aspect ratio is greater than the reference aspect ratio. If the learning image contains a corresponding object (yes in step S47), the processing unit 14 randomly sets N (N is an integer greater than 2) viewpoints for the learning image (step S48). The processing unit 14 may set N viewpoints using the method of embodiment 1. An example of N is 2.

[0117] Next, the processing unit 14 generates N processed images corresponding to the N viewpoints (step S49 ). The processing unit 14 may generate N processed images by executing a viewpoint conversion process so as to change the default viewpoint to the set N viewpoints.

[0118] When the learning image does not include an object whose size is greater than or equal to the reference size and whose aspect ratio is greater than or equal to the reference aspect ratio (No in step S47), the processing unit 14 randomly sets M (M is an integer greater than or equal to 1 and less than N) viewpoints for the learning image (step S50). An example of M is 1. The method of randomly setting viewpoints is the same as that in the first embodiment.

[0119] Next, in step S51 , the processing unit 14 generates M processed images corresponding to the M viewpoints. The processing unit 14 may generate M processed images by executing the viewpoint conversion process so as to change the default viewpoint to the set M viewpoints.

[0120] Next, the processing unit 14 determines whether a given number of learning images have been acquired from the learning image database 23 (step S52). If a given number of learning images have been acquired (yes in step S52), the process ends. On the other hand, if a given number of learning images have not been acquired (no in step S52), the process returns to step S45, and the learning image of the next processing object is acquired from the learning image database 23.

[0121] Thus, according to Embodiment 3, when it is determined that an object with an easily deformable shape, such as a vertically long object, a horizontally long object, or a large-sized object, is included in the learning image, more processed images are generated compared to a case where it is determined that the object is not included. Therefore, it is possible to efficiently generate a learning image that can improve the sensing accuracy of such an object.

[0122] (Implementation method 4)

[0123] In the fourth embodiment, processed images are generated that include more objects that are difficult to sense in the learning model 22. In the fourth embodiment, the same reference numerals are given to the same components as those in the first to third embodiments, and the description thereof is omitted. Figure 1 To explain. Fig.11 4 is a flowchart showing an example of the processing of the image processing device 1 in the fourth embodiment. The processing of steps S71 and S72 is similar to that of Figure 5 Steps S1 and S2 are the same, so the description is omitted. In step S73, the verification unit 13 calculates the sensing accuracy of the object in each verification image according to the classification of the object. For example, when the classification of the object to be sensed includes a sofa, a ceiling light, and a door, the sensing accuracy of each of the sofa, the ceiling light, and the door is calculated.

[0124] Next, in step S74, the verification unit 13 determines whether there is an object belonging to a classification whose sensing accuracy is below the threshold. Hereinafter, an object belonging to a classification whose sensing accuracy is below the threshold is referred to as a specific object. When it is determined that there is a specific object (Yes in step S74), the processing unit 14 obtains a data set of learning images including a given number of learning images from the learning image database 23, and the given number of learning images includes the specific object (step S75). On the other hand, when it is determined that there is no specific object (No in step S74), the processing ends.

[0125] Next, in step S76, the processing unit 14 sets a viewpoint for the learning image. For example, the processing unit 14 may randomly set a viewpoint as in the first embodiment, or may set the midpoint of the longest interval as the viewpoint as in the second embodiment.

[0126] Next, in step S77, the processing unit 14 applies viewpoint conversion processing to each learning image so that the default viewpoint becomes the set viewpoint, thereby generating a processed image (step S77). For example, the processing unit 14 may generate a processed image by applying the viewpoint conversion processing shown in embodiment 1 or embodiment 2.

[0127] As described above, according to the fourth embodiment, since a learning image including an object that is difficult for the learning model to sense is generated, the learning model can be made to learn so that the sensing accuracy of the object is improved.

[0128] (Implementation method 5)

[0129] In Embodiment 5, object detection processing is performed on an omnidirectional image using rule-based object detection processing, and the omnidirectional image subjected to the object detection processing is processed. In Embodiment 5, the same reference numerals are given to the same components as in Embodiments 1 to 4, and description thereof is omitted.

[0130] Fig.12 This is a block diagram showing an example of the configuration of an image processing device 1A in the fifth embodiment. Fig.12 and Figure 1 The differences are that the candidate image database 31 is stored in the memory 20 instead of the verification image database 21, and the processor 10 has a sensing unit 12A, a verification unit 13A and a processing unit 14A instead of the sensing unit 12, the verification unit 13 and the processing unit 14.

[0131] The candidate image database 31 stores candidate images that are candidates for learning of the learning model 22. The candidate images are omnidirectional images associated with correct answer labels, similarly to the verification images.

[0132] The sensing unit 12A applies rule-based object sensing processing to the candidate image acquired by the acquisition unit 11, thereby sensing an object from the candidate image. The so-called rule-based object sensing processing is equivalent to a process of sensing an object from an image without using a learning model obtained by machine learning. An example of rule-based object sensing processing is pattern matching, a process of sensing an object based on the shape of an edge contained in an edge-detected image, and the like. In addition, the classification of the object to be sensed is predetermined. Therefore, the template used in the pattern matching is a template corresponding to the classification of the object to be sensed. The sensing unit 12A calculates the similarity by classification by applying the template by classification to the candidate image.

[0133] Verification unit 13A calculates the similarity calculated by sensing unit 12A as the sensing accuracy of the object sensing process and determines whether the sensing accuracy is lower than a threshold. If the similarities of all the categories are lower than the threshold, verification unit 13A may determine that the sensing accuracy is lower than the threshold.

[0134] When it is determined that the sensing accuracy calculated by the verification unit 13A is lower than the threshold value, the processing unit 14A processes the candidate image so that the deformation of the object included in the candidate image becomes larger.

[0135] The output unit 15 stores the processed image processed by the processing unit 14A in the learning image database 23. This allows the learning model 22 to learn the processed image obtained by processing the candidate image.

[0136] Fig.13 This is a flowchart showing an example of processing of the image processing device 1A in Embodiment 5. First, in step S101 , the acquisition unit 11 acquires a data set of candidate images from the candidate image database 31 .

[0137] Next, the sensing section 12A applies rule-based object sensing processing to each candidate image included in the acquired data set of candidate images, thereby sensing an object from the candidate image (step S102 ).

[0138] Next, in step S103 , the verification unit 13A calculates the similarity calculated when the sensing unit 12A senses the object as the sensing accuracy.

[0139] Next, in step S104, the verification unit 13A determines whether the sensing accuracy is less than a threshold value. If the sensing accuracy is less than the threshold value (yes in step S104), the processing unit 14A sets a viewpoint for the candidate image (step S105). For example, the processing unit 14A may randomly set a viewpoint as shown in Embodiment 1, or may set the midpoint of the longest interval as a viewpoint as shown in Embodiment 2. If the sensing accuracy is greater than the threshold value (no in step S104), the process ends.

[0140] Next, in step S106, the processing unit 14A applies a viewpoint conversion process to the candidate image so that the default viewpoint becomes the set viewpoint, thereby generating a processed image (step S106). For example, the processing unit 14A may generate the processed image by applying the viewpoint conversion process described in Embodiment 1 or Embodiment 2. The processed image is stored in the learning image database 23.

[0141] As described above, according to the fifth embodiment, since a candidate image determined to be an object with low sensing accuracy by the rule-based object sensing process is processed, a processed image for learning including the object can be generated.

[0142] The present disclosure can adopt the following modified examples.

[0143] (1) In Embodiments 1 to 4, the omnidirectional image to be processed is a learning image stored in the learning image database 23 , but may be a verification image.

[0144] (2) The method of acquiring a learning image including a specific object from the learning image database 23 described in the fourth embodiment can also be applied to the first to third embodiments.

[0145] (3) In the above embodiments, a construction site is exemplified as the site, but the present disclosure is not limited thereto, and a manufacturing site, a logistics site, a distribution site, a farmland, a construction site, a retail site, an office, a hospital, a commercial facility, a nursing facility, etc. may also be used as the site.

[0146] Industrial Applicability

[0147] According to the present disclosure, it is useful in the technical field of object sensing from omnidirectional images.

Claims

1. An image processing method is an image processing method in a computer. Get an image composed of omnidirectional images, performing object sensing processing of sensing an object from the acquired image, calculating the sensing accuracy of the object in the object sensing process, Based on the sensing accuracy, the image is processed so that the deformation of the object contained in the image becomes larger, The processed image is output.

2. The image processing method according to claim 1, wherein: The image is an image composed of an omnidirectional image corresponding to the correct label of the object. Calculating the sensing accuracy based on the correct answer label, The processing is performed when the sensing accuracy is lower than a threshold value.

3. The image processing method according to claim 1 or 2, wherein: The image includes a first image and a second image different from the first image. The sensing accuracy is a sensing accuracy with respect to a sensing result when the first image is input into a learning model that has been learned in advance for executing the object sensing process. The image processing is performed on the second image.

4. The image processing method according to claim 3, wherein: Furthermore, the processed image is used to enable the learning model to learn.

5. The image processing method according to claim 3, wherein: The sensing accuracy is calculated according to the classification of the object. The second image is an image including an object determined in the first image to have the sensing accuracy being equal to or less than a threshold value.

6. The image processing method according to claim 1 or 2, wherein: The processing of the image includes changing a default viewpoint of the image to a randomly set viewpoint.

7. The image processing method according to claim 1 or 2, wherein: The processing of the image includes: determining an interval between two binding frames having the longest distance among a plurality of correct answer labels set in the image, and setting a viewpoint of the image at a midpoint of the interval.

8. The image processing method according to claim 1 or 2, wherein: The processing of the image comprises: determining whether the image contains an object whose aspect ratio and size at least one of which exceeds a reference value; and When it is determined that an object exceeding the reference value is included, the number of processed images is increased compared to when it is determined that the object is not included.

9. The image processing method according to claim 1 or 2, wherein: The object sensing process is a rule-based object sensing process, The image processing includes performing the image processing on the image that has been subjected to the object sensing process.

10. The image processing method according to claim 1 or 2, wherein: The processing of the image includes: processing the image by performing a viewpoint conversion process so that the deformation of the object becomes larger, The viewpoint conversion process comprises: Projecting the image onto a unit sphere; Setting a new viewpoint based on the projected image that has been projected; and The projected image is expanded along a plane so that the new viewpoint becomes the center.

11. An image processing device comprising a processor, The processor performs the following processing: Get an image composed of omnidirectional images, performing object sensing processing of sensing an object from the acquired image, calculating the sensing accuracy of the object in the object sensing process, Based on the sensing accuracy, the image is processed so that the deformation of the object contained in the image becomes larger, The processed image is output.

12. An image processing program causing a computer to execute the image processing method according to claim 1 or 2.

Images