Image processing method and image processing system

Abstract

Providing a technology for accurately detecting objects from equirectangular images. [Solution] In System 1, an equirectangular input image 10 is mapped onto a sphere, and multiple partial images are extracted so that parts of adjacent regions overlap according to settings made in advance by the user, and object detection is performed on these. Subsequently, the vertex coordinates of the mask contour are obtained from the mask image, which is converted to equirectangular format by masking the area outside the rectangular frame surrounding the region where the object was detected, and the vertex coordinates corresponding to multiple rectangular frames in which the same object was detected are integrated into one, and an output image 20 with the integrated vertices added to the input image 10 and its detailed data 21 (detection class, vertex coordinates, confidence level) are output. Since the user can set the extraction of partial images based on the object to be detected, it is possible to extract partial images that are easy to detect objects in, and thus enable accurate object detection.

Description

Technical Field

[0001] The present invention relates to a method for detecting an object from an equirectangular format image, and a system to which this method is applied.

Background Art

[0002] In recent years, the use of BIM models in FM has attracted attention in order to improve the efficiency of facility management (hereinafter referred to as "FM") that occupies most of the building life cycle. However, there is no BIM model for buildings constructed only from 2D drawings. Therefore, various studies have been made using 3D point clouds, images, etc. obtained from buildings in order to generate BIM models of existing buildings.

[0003] A 3D scanner often used at construction sites is equipped with a 360-degree camera that can photograph the entire surrounding area at once. 360-degree images taken by a 360-degree camera are generally output in an equirectangular format. In order to generate a BIM model based on an image, it is necessary to detect an object to be modeled from the image. However, general pre-trained models used for object detection do not support equirectangular format images. In addition, creating a pre-trained model corresponding to an equirectangular format image takes a huge amount of labor and time, including preparing training data. Therefore, in order to detect an object shown in an equirectangular format image, usually, the equirectangular format image is converted into a cube map image projected onto six faces (front, back, top, bottom, left, and right faces) of a cube, and images of each face are input into a general pre-trained model.

[0004] However, when converted to a cubemap image, objects projected across the boundaries of each face appear as fragmented images in each face, making detection difficult. Therefore, various methods have been employed to address this challenge. For example, Patent Document 1 discloses a method in which an equirectangular image is converted to a cubemap image, and from the multiple faces constituting the cubemap image, a panoramic image is created by arranging them as they are, and another panoramic image is created by horizontally shifting the entire image to wrap around it. Object detection is then performed on each panoramic image, and the object detection results are combined. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 7320660 [Overview of the project] [Problems that the invention aims to solve]

[0006] According to the prior art described above, even if an object is interrupted at the boundary in one panoramic image, that object will not be interrupted in the other panoramic image. Therefore, by combining the object detection results, the desired detection result can be obtained, and it is thought that the above-mentioned problems related to cubemap images can be addressed.

[0007] However, cubemap images are created by extracting six images from an equirectangular image with a 90-degree field of view and a 1:1 aspect ratio. Because the field of view is fixed, the size of objects in the extracted images cannot be changed, making it difficult to correctly detect small objects. This is a major challenge not only in the construction industry mentioned above, but also in all fields that require object detection from equirectangular images.

[0008] Therefore, the present invention aims to provide a technology for accurately detecting objects from equirectangular images. [Means for solving the problem]

[0009] To solve the above problems, the present invention employs the following image processing method and image processing system to which the same is applied. Note that the following statements in parentheses are merely examples, and the present invention is not limited thereto.

[0010] In other words, the image processing method of the present invention is an image processing method for detecting an object from an equirectangular image, and includes a setting step of setting up a portion of an image mapped to a sphere to be cut out as a rectangular partial image; a cutting step of mapping the image to a sphere and cutting out a plurality of partial images such that portions of adjacent regions overlap according to the setting; a detection step of detecting an object contained in the partial image; a result conversion step of creating a mask image by masking the area outside the rectangular frame surrounding the region in which the object was detected in the partial image and converting it to equirectangular format, and obtaining the vertex coordinates of the contour corresponding to the rectangular frame from the mask image; and a result integration step of integrating the vertex coordinates of the plurality of mask images according to a predetermined rule.

[0011] According to this embodiment of the image processing method, an equirectangular image is mapped onto a sphere, object detection is performed on multiple sub-images extracted from this sphere, and the detection results are converted back into an equirectangular image and integrated. Therefore, both the input and output can be in equirectangular format. Furthermore, since object detection is performed on sub-images, a pre-trained model compatible with common image formats can be used, enabling accurate and efficient object detection.

[0012] Preferably, in the image processing method of the above embodiment, the setting step is to set the field of view from the center of the sphere to a width such that a predetermined object to be detected fits within a single partial image.

[0013] According to this embodiment of the image processing method, the field of view (FOV) from the center of the sphere is set to a width such that a predetermined object (for example, the largest of the objects to be detected) fits within a single partial image. This makes it possible to extract a partial image that is easy to perform object detection on, and to perform object detection from the partial image with high accuracy.

[0014] Preferably, in any of the image processing methods described above, a first angle for rotating the line of sight from the center of the sphere vertically and a second angle for rotating it horizontally are further set. In the cropping step, partial images are successively cropped by rotating the line of sight horizontally by two angles at a time while keeping the vertical angle fixed, and when the cumulative rotation angle in the horizontal direction reaches 360 degrees, the line of sight is rotated vertically by one angle, and while keeping this angle fixed, it is rotated horizontally by two angles at a time to successively crop partial images, and partial images are cropped from all directions in this manner.

[0015] According to this image processing method, by appropriately setting a first angle (the increase in pitch per rotation) and a second angle (the increase in yaw per rotation) related to the rotation of the line of sight from the center of the sphere, and then successively cutting out partial images from different angles while changing the direction of the line of sight according to these settings, it is possible to obtain partial images in which the object is captured at various angles.

[0016] This allows individual objects to be detected to be captured in a partial image without interruption at any angle, enabling accurate object detection from partial images. Furthermore, some pre-trained models tend to have a higher detection rate when objects are captured at a specific angle, and by targeting partial images in which objects are captured at various angles, the accuracy of object detection when using such pre-trained models can be improved.

[0017] Preferably, in the image processing method of any of the above embodiments, in the cropping step, when the angle in the vertical direction of the line of sight is a predetermined angle, a straight line is detected from the cropped portion image at this angle, and the direction of the line of sight is rotated horizontally by an angle corresponding to the slope of the straight line to further crop a portion image in which the straight line is extended vertically.

[0018] Some pre-trained models tend to detect specific objects (e.g., pipes) from partial images more accurately when the object is oriented vertically than when it is tilted. According to this embodiment of the image processing method, when the vertical angle of the line of sight (Pitch) is a predetermined angle (e.g., -90 degrees), the line of sight is rotated horizontally by an angle corresponding to the tilt of the line detected in the partial image, and then the partial image is further cropped. This makes it possible to crop a partial image in which the specific object is stretched vertically, thereby improving the object detection accuracy when using the aforementioned pre-trained model.

[0019] Preferably, in any of the image processing methods described above, in the result integration step, when multiple mask images are integrated and contours overlap, only the vertex coordinates of the contour corresponding to the rectangular frame with the highest confidence rate are retained, taking into account the class of the object detected in the rectangular frame.

[0020] According to this embodiment of the image processing method, multiple detection results for the same object obtained by performing object detection on multiple cropped images can be integrated into one. As a result, in the final equirectangular output image, only the vertices obtained by the integration can be displayed, and the user can easily understand the detection results from the output image. [Effects of the Invention]

[0021] As described above, according to the present invention, object detection from equirectangular images can be performed with high accuracy. [Brief explanation of the drawing]

[0022] [Figure 1] It is a block diagram showing the configuration of the image processing system 1. [Figure 2] It is a diagram for explaining the setting items related to the extraction of partial images. [Figure 3] It is a drawing substituting for a photograph for explaining the method of adjusting the FOV. [Figure 4] It is a flowchart showing an example of the procedure of the image extraction process. [Figure 5] It is a diagram for explaining a specific process. [Figure 6] It is a drawing substituting for a photograph showing an example of a partial image cut out from an input image. [Figure 7] It is a drawing substituting for a photograph showing an example of the result of object detection for a partial image. [Figure 8] It is a drawing substituting for a photograph for explaining the detection result conversion process. [Figure 9] It is a diagram for explaining the detection result integration process. [Figure 10] It is a diagram showing an example of the integration of the detection results of the illumination class. [Figure 11] It is a drawing substituting for a photograph showing the result finally obtained by the image processing system 1.

Embodiments for Carrying Out the Invention

[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments are a preferred example of an image processing system to which the image processing method of the present invention is applied, and the present invention is not limited to this example. Also, due to the nature of the invention, it is difficult to illustrate the differences in detection results according to the way the object to be detected appears in the photograph, so photographs are used instead of the drawings in various parts of the drawings.

[0024] 〔Configuration of Image Processing System〕 FIG. 1 is a block diagram showing the configuration of an image processing system 1 according to an embodiment. Image processing system 1 is a system that performs processing to detect objects from equirectangular images. Image processing system 1 is broadly composed of a setting unit 2, an image extraction unit 3, a detection control unit 4, an object detection unit 5, a detection result conversion unit 6, and a detection result integration unit 7.

[0025] The setting unit 2 generates a setting screen for various settings necessary for extracting a portion of an equirectangular input image 10, which is input to the image processing system 1, as a rectangular sub-image. The setting unit 2 accepts and stores setting values ​​from the user via the setting screen.

[0026] The image extraction unit 3 maps the equirectangular input image 10 onto a sphere and then extracts multiple partial images such that parts of adjacent regions overlap, according to settings previously made by the user.

[0027] The detection control unit 4 inputs the partial images extracted from all directions by the image extraction unit 3 to the object detection unit 5 and detects objects in the partial images. The object detection unit 5 is a pre-trained model that corresponds to a general image format and detects objects contained in the partial images. The object detection unit 5 can be a pre-trained model that has been trained to detect objects of a specific category (for example, a model trained using YOLO v5) or a general-purpose zero-shot model for object detection (for example, Grounding DINO). If the object detection unit 5 is a zero-shot model, the detection control unit 4 inputs text information specifying the object to be detected along with the partial image to the object detection unit 5, and the object detection unit 5 detects the object specified by the text information from the partial image.

[0028] The detection result conversion unit 6 converts the object detection result from the object detection unit 5 into an equirectangular format by masking the area outside the rectangular frame surrounding the region where the object was detected in the partial image, and then obtains the vertex coordinates of the contour corresponding to the rectangular frame from this mask image.

[0029] The detection result integration unit 7 integrates the vertex coordinates corresponding to the contours of the rectangular frames in which the same object was detected for multiple equirectangular mask images created by the detection result conversion unit 6, according to a predetermined rule. In addition to an output image 20 with the detection result, which reflects the result of the vertex coordinate integration on the input image 10, it also outputs detailed data 21 regarding the detection result in a format such as CSV.

[0030] The details of each process performed from the time the input image 10 is input to the image processing system 1 until the output image 20 and detailed data 21 are output will be described in detail later with reference to a separate diagram.

[0031] [Settings related to cropping partial images] Figure 2 illustrates the settings related to cropping a portion of an image. As described above, the equirectangular input image 10 is mapped onto a sphere, and a portion of it is then extracted as a rectangular sub-image. In this regard, settings are made in advance regarding how the line of sight from the center of the sphere relative to the sphere is moved, the width of the field of view, and the size of the sub-image to be extracted.

[0032] Figure 2(A) shows the vertical and horizontal rotation angles of the line of sight when viewing an object P in the input image 10 mapped onto a sphere from the center O of the sphere. For the movement of the line of sight from the center of the sphere, the rotation ranges of Pitch and Yaw (the angle rotated and the increase in angle per change) are set.

[0033] Pitch is the angle at which the line of sight moves up and down, and this corresponds to angle α in the figure. Pitch can be changed within a range from -90 degrees, which is the angle when looking straight up, to 90 degrees, which is the angle when looking straight down. From the inventors' verification results, it has been found that in order to detect equipment such as windows and lighting fixtures from equirectangular images taken of a room, setting the rotation range of Pitch (incP, described later) to about 30 to 45 degrees makes it easier to generate seamless partial images of objects.

[0034] Yaw is the angle of movement of the gaze from side to side, and this corresponds to angle β in the figure. Yaw can be changed within a range from 0 degrees, which is the angle when looking straight ahead, to 360 degrees, which is the angle when rotating 360 degrees horizontally and returning to the front. From the inventors' verification results, it has been found that for the above purpose, setting the rotation range of Yaw (incY, described later) to about 30 to 45 degrees makes it easier to generate seamless partial images of objects.

[0035] Figures 2 (B-1) and (B-2) show the difference in how object P appears (how it is captured) in a partial image depending on the field of view from the center O of the sphere. The field of view from the center of the sphere is set using the FOV (field of view). The width w and height h (both pixel values) of the partial image to be extracted are also set.

[0036] As shown in Figure 2 (B-1), in the partial image CP1 cropped with width w × height h when the field of view is γ1, object P appears small. In contrast, as shown in Figure 2 (B-2), in the partial image CP2 cropped with width w × height h when the field of view is smaller than γ1 (γ2), object P appears larger than in partial image CP1. Thus, in partial images cropped to a certain size, objects appear larger as the FOV decreases, and smaller as the FOV increases. Taking this into consideration, when setting the FOV, the largest object to be detected (hereinafter referred to as "target object") should be sized to fit within a single partial image.

[0037] Figure 3 is a photographic substitute diagram illustrating how to adjust the FOV. When setting up the cropping, first confirm that the object to be detected is visible in the equirectangular input image 10. The input image 10 shown in Figure 3 (A) is a 360-degree image taken inside a room, and it is necessary to detect equipment such as windows and lighting fixtures from this input image. Multiple lighting fixtures are visible at the top of this input image, and a large window frame WF is visible in the center.

[0038] Once it is confirmed that the object to be detected is present, the next step is to adjust the FOV to an appropriate size while checking how it appears in the partial image. Specifically, the FOV should be adjusted so that the largest object to be detected (for example, the entire window frame) fits within one partial image. Figure 3(B) shows the difference in how the window frame appears in the partial image depending on the FOV size. As mentioned above, the cubemap image is created by extracting images of the six faces of a cube from an equirectangular image with an FOV of 90 degrees. However, when the partial image is taken with the same 90-degree FOV, it can be seen that the entire window frame does not fit within the partial image, and a large portion of it is cut off. Therefore, if the FOV is increased by 10 degrees at a time, the entire frame just barely fits within the partial image at an FOV of 120 degrees, but considering the ease of detection, it is preferable to have a little margin. When the FOV is set to 130 degrees, the entire window frame fits sufficiently within the partial image. Therefore, in this case, the FOV should be set to 130 degrees.

[0039] In this example, the appropriate FOV is determined by checking the input image. However, it is also possible to configure the system to have multiple fixed FOV levels (for example, three levels: low FOV, medium FOV, and high FOV), allowing the user to select and set one of these levels depending on the object being detected. In this case, high FOV would be selected when detecting objects such as window frames that are likely to appear relatively large in partial images, low FOV would be selected when detecting objects such as lighting fixtures that are likely to appear relatively small, and medium FOV would be selected when detecting objects of an intermediate size. This configuration eliminates the need to constantly adjust the FOV and check how it appears in partial images, allowing the user to set the FOV efficiently.

[0040] [Image cropping process] Figure 4 is a flowchart showing an example of the image extraction process performed by the image extraction unit 3. The process will be explained below in accordance with this example.

[0041] Step S1: The image extraction unit 3 sets the initial value of Pitch to -90 degrees (Pitch=-90°) and the initial value of Yaw to 0 degrees (Yaw=0°).

[0042] Step S2: The image extraction unit 3 performs a specific process when the pitch is -90 degrees, that is, when the line of sight is pointing straight up. The details of the specific process will be described later with reference to another diagram.

[0043] Step S3: The image extraction unit 3 adds the rotation amount incP to Pitch (Pitch = Pitch + incP), thereby rotating the line of sight vertically by the rotation amount incP from the current orientation, and resetting Yaw to 0 degrees (Yaw = 0°).

[0044] Steps S4-S6: The image extraction unit 3 checks whether the pitch is less than 90 degrees. If the pitch is less than 90 degrees, i.e., the line of sight does not reach directly below (Step S4: Yes), it extracts a portion of the image with width w × height h at the current position (Step S5), and then rotates the line of sight by rotation incY from the current direction by rotation incY by adding rotation incY to Yaw (Yaw = Yaw + incY) (Step S6). On the other hand, if the pitch is 90 degrees or more, i.e., the line of sight reaches directly below (Step S4: No), the image extraction unit 3 terminates the image extraction process.

[0045] Step S7: The image extraction unit 3 checks whether Yaw is less than 360 degrees. If Yaw is less than 360 degrees, i.e., the line of sight has not yet completed one rotation in the left-right direction (Step S7: Yes), it returns to Step S5 and performs image extraction (Step S5) and Yaw addition (Step S6) again. On the other hand, if Yaw is 360 degrees or more, i.e., the line of sight has completed one rotation in the left-right direction (Step S7: No), the image extraction unit 3 returns to Step S3, performs Pitch addition and Yaw reset (Step S3), and then repeats the subsequent steps.

[0046] In this way, the extraction of partial images from the sphere to which the input image is mapped begins with the extraction of the region directly above, and while gradually changing the Pitch and Yaw, multiple partial images are extracted so that parts of adjacent regions overlap, and the process ends when the region directly below is reached.

[0047] The above procedure example is merely an example and can be modified as needed. For example, in the above procedure example, the initial value of Pitch is set to -90 degrees, and increments of incP are added repeatedly until 90 degrees are reached, while the partial image extraction and Yaw change are repeated. Alternatively, the initial value of Pitch could be set to 90 degrees, and increments of incP are subtracted repeatedly until -90 degrees is reached. In this case as well, the specific processing will be executed when Pitch is -90 degrees.

[0048] Figure 5 illustrates a specific process performed during the image extraction process (step S2 in Figure 4). The details of this process will be explained below, following the example procedure shown in Figure 5(A).

[0049] Step S11: The image extraction unit 3 extracts a portion of the image with width w × height h at the current position. At this time, it is assumed that the extracted portion of the image shows the piping PP installed on the ceiling, for example, as shown in (B-1) in Figure 5.

[0050] Step S12: The image extraction unit 3 attempts to detect straight lines from the partial image extracted in step S11. Known methods such as the Hough transform can be used to detect straight lines from the partial image. In the illustrated example, four straight pipes PP are visible in the partial image, and their shapes are detected as straight lines.

[0051] Steps S13-S15: The image extraction unit 3 checks whether a straight line has been detected. If no straight line is detected, i.e., if no linear object is captured in the partial image (Step S13: No), the image extraction unit 3 terminates the identification process. On the other hand, if a straight line is detected, i.e., if a linear object is captured in the partial image (Step S13: Yes), the unit identifies the slope of the straight line detected in Step S12 (Step S14) and changes the Yaw angle so that the straight line extends vertically in the partial image (Step S15). Specifically, the image extraction unit 3 changes the Yaw angle so that the angle θ in the figure becomes 0 degrees.

[0052] Step S16: The image extraction unit 3 extracts a partial image again at the position where Yaw was changed in step S15 above. At this time, as shown in (B-2) in Figure 5, a partial image is extracted in which the pipe PP is shown stretched in the vertical direction.

[0053] Verification by the inventors has shown that when detecting pipes from a partial image, the detection rate is higher when the pipes are oriented vertically than when they are tilted. Also, pipes are installed in specific locations such as ceilings. Therefore, in this embodiment, when the pitch is -90 degrees, a specific process is performed to extract a partial image in which the pipes are oriented vertically.

[0054] In this embodiment, the specific processing is performed only when the Pitch is -90 degrees, but it may also be performed at other angles (for example, 90 degrees). Alternatively, in addition to Pitch and Yaw, Roll, another parameter of the Euler angle, may be adjusted as appropriate before extracting a partial image.

[0055] Figure 6 is a photographic substitute drawing showing an example of a partial image extracted from an input image. Figure 6(A) shows an example of an equirectangular input image 10 input to the image processing system 1, and Figure 6(B) shows four examples of partial images extracted from the input image shown in Figure 6(A). In the illustrated examples, the size of the input image 10 is 8000 × 4000 pixels, and the size of the partial images is 600 × 600 pixels, but due to space limitations in the drawings, the input image and partial images are shown scaled down to different scales (the same applies to the other drawings).

[0056] The illustrated partial images were extracted from the input image by gradually changing the Pitch and Yaw rotation ranges, with the Pitch rotation range incP and Yaw rotation range incY set to 45 degrees each. From Figure 6(A), it can be seen that the input image shows a window frame WF and a television TV. In the upper right partial image of Figure 6(B), part of the window frame WF is cut off, but the entire window frame WF is included in the upper left partial image. Also, only part of the television TV is visible in the lower right partial image, but the entire television TV is included in the upper right and lower left partial images. In this way, by changing the Pitch and Yaw rotation ranges by the predetermined FOV, partial images are extracted with overlapping portions of adjacent areas, allowing each object to be included in the partial image without interruption at any angle.

[0057] [Results of object detection from partial images] Figure 7 is a photographic substitute diagram showing an example of the results of object detection performed by the object detection unit 5 on a partial image. The illustrated image shows the results of object detection performed on a partial image capturing adjacent areas within the same room. Above the rectangular frame (more precisely, Bounding Box) surrounding the detected object, the detection class and its confidence score (the probability that the object belongs to the detection class) are indicated as a percentage. For example, the notation "lamp2:89.3%" attached to the rectangular frame of lighting LM1 in Figure 7 (A) indicates that lighting LM1 was determined to belong to the "lamp2" class with a probability of "89.3%". The trained model used for this detection was trained after classifying various equipment in the room into multiple classes according to their type, and "lamp2" is one of the multiple classes related to lighting.

[0058] The partial image in Figure 7(A) shows four lights LM1-4. Three of these lights, LM1, 3, and 4, were determined to belong to the "lamp2" class with a confidence level of over 80%, meaning they were correctly detected as lights. In contrast, light LM2 was determined to belong to the "smoke_detector" class, meaning it was falsely detected as a smoke detector.

[0059] Furthermore, due to the pitch and yaw when a partial image is extracted, the same lighting may be captured at different angles, resulting in varying detection rates depending on how it is captured, and in some cases, detection may not occur at all. For example, lighting LM2 is visible in all partial images from (A) to (D) in Figure 7, but in Figure 7 (C), it is captured in a very ambiguous manner, and therefore it was not detected by the object detection unit 5.

[0060] Thus, in multiple partial images, the same object is detected correctly or incorrectly with different confidence levels. Therefore, in subsequent processing, these detection results are converted to equirectangular format, and then the multiple detection results for the same object are merged.

[0061] [Detection result conversion process] Figure 8 is a photographic substitute diagram illustrating the detection result conversion process performed by the detection result conversion unit 6.

[0062] In the detection result conversion process, the coordinates of the rectangular frame are first obtained from the detection result, and a mask image is created by filling the inside of the rectangular frame with white and the rest with black. Specifically, in the case of a detection result like that shown in Figure 8(A), a mask image like that shown in Figure 8(B) is created, in which the inside of the rectangular frame of illumination LM1 is filled with white and the rest with black.

[0063] Next, based on the Pitch, Yaw, and FOV values ​​obtained when the partial image was extracted, this mask image is converted to equirectangular format. Converting the mask image shown in Figure 8(B) to equirectangular format yields the equirectangular mask image shown in Figure 8(C).

[0064] Then, the contour of the foreground (the area of ​​white pixels) is obtained from the equirectangular mask image, and the vertex coordinates corresponding to this contour are obtained. For example, from the mask image shown in Figure 8(C), the coordinates of vertices a, b, c, and d of the foreground are obtained, as shown in Figure 8(D). Note that in Figure 8(D), each vertex is exaggerated for easier viewing. In addition, existing image processing libraries (for example, the findContours function of OpenCV) can be used to extract the contour of the mask.

[0065] [Integration of detection results] Figure 9 illustrates the detection result integration process performed by the detection result integration unit 7.

[0066] As described above, a single object appears in multiple partial images, and is detected correctly or incorrectly from each partial image with different confidence levels. Therefore, when the detection result transformation process (Figure 8) is performed on multiple detection results, the vertex coordinates of the masks corresponding to multiple rectangular frames for that object are obtained. Therefore, a detection result integration process is performed to retain only the vertex coordinates of the mask contour corresponding to the rectangular frame with the highest confidence level that was correctly detected. The Non-Maximum Suppression (NMS) method is used for integrating the results.

[0067] In the detection result integration process, first, the vertex coordinates of masks of the same class are merged using NMS. For example, as shown in the left figure of Figure 9(A), assume that multiple rectangular frames overlap a single object (a watermelon in the example shown). In this case, among the rectangular frames detected as belonging to the same class, the rectangular frame with the highest confidence level (the "92%" rectangular frame in the example shown) is used as the reference, and other rectangular frames whose degree of overlap (IoU) with this rectangular frame exceeds a certain level are considered duplicates and removed. As a result, as shown in the right figure of Figure 9(A), the vertex coordinates of masks of the same class are integrated into the vertex coordinates of the mask corresponding to the rectangular frame with the highest confidence level.

[0068] In the example in Figure 9(A), since no rectangles are mistakenly identified as belonging to a different class, it is sufficient to simply merge the vertex coordinates of masks of the same class. In contrast, as shown in the left figure of Figure 9(B), if multiple rectangles overlapping a single object include rectangles that are mistakenly identified as belonging to a different class (for example, the rectangle mistakenly identified as a beach ball, shown by the dashed line in the figure), simply merging by the same class will leave the detection results of the misdetection, as shown in the middle figure of Figure 9(B). Therefore, in such cases, the vertex coordinates of the masks are further merged using NMS for all classes. This makes it possible to retain only the vertex coordinates of the masks corresponding to the rectangles with the highest confidence level that were correctly detected, as shown in the right figure of Figure 9(B).

[0069] Figure 10 shows an example of integrating the detection results for lighting classes. By integrating the multiple detection results of the illumination class shown in Figure 10(A) (vertex coordinates obtained from multiple mask images of the illumination class obtained in the detection result conversion process (Figure 8)), the result shown in Figure 10(B) is obtained. In Figure 10(B), the multiple detection results are shown as an overlay, with areas where the rectangular frames overlap more appearing brighter and areas where the overlaps less appearing darker.

[0070] [Final output] Figure 11 shows the final results obtained by the image processing system 1. Once the detection results are integrated, the image processing system 1 (detection result integration unit 7) outputs an image with the detection results. As shown in Figure 11 (A), the output image 20 with the detection results has the integrated vertices of the rectangular frames surrounding each detected object (in the illustrated example, the four lights LM1-LM4) added to the image 10 (Figure 6 (A)) input to the image processing system 1. For the sake of explanation, only the detection results for the lighting class are shown in the illustrated example, but the actual output image will display the detection results for all classes corresponding to the object being detected.

[0071] In addition to the output image 20 with the detection results, detailed data 21 regarding the detection results (specifically, the detection class, vertex coordinates, and confidence level for each object) is output in a format such as CSV. Further information may be displayed on the output image 20, or further information may be added to the detailed data 21. For example, the vertices displayed on the output image 20 may be displayed in multiple colors according to the confidence level of the detection of the object enclosed by the vertices (for example, vertices may be displayed in blue if the confidence level is 90% or higher, green if it is 60% or higher, yellow if it is 30% or higher, and red if it is less than 30%).

[0072] As described above, the image processing system 1 and method of the above-described embodiment have the following effects. (1) Multiple partial images are extracted from the equirectangular input image 10 to perform object detection, and the multiple detection results are converted to equirectangular format and integrated to output an output image 20 that reflects the detection results. Thus, the user can input an equirectangular image and obtain the object detection results in an equirectangular image.

[0073] (2) When extracting a portion of an input image mapped onto a sphere as a partial image, the FOV can be adjusted so that the largest object to be detected fits into one partial image. This makes it possible to extract a partial image that is easier to detect, thereby improving the accuracy of object detection.

[0074] (3) By varying the Pitch and Yaw by a fixed amount and extracting multiple sub-images so that parts of adjacent regions overlap, it is possible to obtain sub-images in which a single object is captured from various angles, and the object can be captured in sub-images without interruption at any angle. Furthermore, when using a zero-shot model for object detection, the detection rate tends to be higher when the object is captured at a specific angle. Therefore, by inputting sub-images in which the object is captured at various angles into the zero-shot model, it is possible to detect a single object from various angles, which contributes to improving the detection accuracy.

[0075] (4) Since object detection is performed on partial images, a general pre-trained model can be used for object detection, saving the enormous amount of time and effort that would be required to create a pre-trained model for equirectangular images, and enabling efficient detection of objects in equirectangular images.

[0076] The present invention can be implemented in various ways without being limited to the embodiments described above.

[0077] In the embodiment described above, the input image size is 8000 x 4000 pixels and the partial image size is 600 x 600 pixels, but the sizes of the input image and partial image are not limited to these. Furthermore, the width w and height h of the partial image do not need to match, and it is possible to set the optimal pixel values ​​depending on the situation.

[0078] In the embodiment described above, the setting unit 2 generates a setting screen and accepts setting values ​​from the user via the setting screen. However, instead, the setting values ​​may be manually stored in a setting file or the like beforehand.

[0079] In the embodiment described above, the image processing system 1 includes a setting unit 2, an image extraction unit 3, a detection control unit 4, an object detection unit 5, a detection result conversion unit 6, and a detection result integration unit 7. However, the division of these functional units 2 to 7 is merely an example, and the functional units may be further subdivided, or some of the functional units may be integrated.

[0080] Furthermore, the numerical values ​​and other figures shown in the embodiments are merely preferred examples and can be modified as appropriate when implementing the present invention. [Explanation of symbols]

[0081] 1. Image Processing System 2 Setting section (setting process) 3 Image cutting section (cutting process, cutting means) 4. Detection Control Unit 5. Object detection unit (detection process, detection means) 6. Detection Result Conversion Unit (Result Conversion Process, Result Conversion Means) 7. Detection Result Integration Unit (Result Integration Process, Result Integration Means) 10 Input Images 20 Output Images

Claims

1. An image processing method for detecting objects from equirectangular images, A setting step involves setting up the process to extract a portion of the image mapped to a sphere as a rectangular sub-image, A cutting step of mapping the aforementioned image onto a sphere and cutting out multiple partial images such that parts of adjacent regions overlap according to the settings, A detection step for detecting an object included in the aforementioned partial image, The process includes creating a mask image by masking the area outside the rectangular frame surrounding the region where an object was detected in the partial image and converting it to equirectangular format, and obtaining the vertex coordinates of the contour corresponding to the rectangular frame from the mask image, and A result integration step of integrating the vertex coordinates in multiple mask images according to a predetermined rule. Image processing methods including [specific details omitted].

2. In the image processing method described in claim 1, In the aforementioned setup process, An image processing method characterized by setting the field of view from the center of a sphere to a size such that a predetermined object to be detected fits within one of the partial images.

3. In the image processing method described in claim 2, In the aforementioned setup process, An image processing method characterized by further setting a first angle for rotating the line of sight from the center of the sphere in the vertical direction and a second angle for rotating it in the horizontal direction.

4. In the image processing method described in claim 3, In the aforementioned cutting process, An image processing method characterized by extracting partial images from all directions by following a process in which the line of sight is rotated by the second angle in the left-right direction while keeping the angle in the vertical direction fixed, and when the cumulative rotation angle in the left-right direction reaches 360 degrees, the line of sight is rotated by the first angle in the vertical direction, and while keeping this angle fixed, it is rotated by the second angle in the left-right direction to extract the partial images one after another.

5. In the image processing method described in claim 4, In the aforementioned cutting process, An image processing method characterized by detecting a straight line from the partial image cut out at a predetermined angle when the angle of the line of sight in the vertical direction is such that the direction of the line of sight is rotated horizontally by an angle corresponding to the inclination of the straight line, and further cutting out a partial image in which the straight line is extended vertically.

6. In the image processing method described in claim 1, In the aforementioned results integration process, An image processing method characterized in that, when multiple mask images are integrated and the contours overlap, only the vertex coordinates of the contour corresponding to the rectangular frame with the highest confidence rate, taking into account the class of the object detected in the rectangular frame, are retained.

7. An image processing system for detecting objects from equirectangular images, A cutting means for mapping the aforementioned image onto a sphere and cutting out a portion of it as a rectangular partial image, according to a predetermined setting, such that portions of adjacent regions overlap, A detection means for detecting an object included in the aforementioned partial image, A result transformation means that creates a mask image by masking the area outside the rectangular frame surrounding the region where an object was detected in the partial image and then converting it to equirectangular format, and obtains the vertex coordinates of the contour corresponding to the rectangular frame from the mask image, A result integration means for integrating the vertex coordinates in a plurality of mask images according to a predetermined rule. An image processing system equipped with [the following features].

Images