Object state evaluation system, object state evaluation method, and object state evaluation program

The object state evaluation system addresses inaccuracies in existing systems by using camera calibration and 3D model generation to assess object states accurately, despite changes in appearance and camera positioning.

JP2024168690APending Publication Date: 2024-12-05HITACHI LTD

Patent Information

Application Number
JP2023085572
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing object evaluation systems struggle to accurately assess the state of objects due to changes in appearance, camera positioning, and occlusion, and require known camera parameters, leading to inaccuracies in depth estimation and texture uniformity.

Method used

An object state evaluation system that includes a processor to generate evaluation images, determine object posture, and match them with predetermined templates, using camera calibration and 3D model generation when necessary, to assess object states.

Benefits of technology

Enables easy and appropriate evaluation of object states by accounting for camera positioning and occlusion, and generating accurate 3D models for precise matching, regardless of camera parameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168690000001_ABST
    Figure 2024168690000001_ABST
Patent Text Reader

Abstract

To evaluate a state of an object easily and appropriately.SOLUTION: An object state evaluation system 10 for evaluating a state of an object in an image includes a processor. The processor is configured to: acquire an image including an object to be evaluated; generate an evaluation image in which the object has a first attitude; and determine whether the object is in a predetermined state, on the basis of the evaluation image and a determination template image in which the object has the first attitude in the predetermined state.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for evaluating the state of an object. [Background technology]

[0002] For example, the appearance of an object analyzed by a video analytics system may change over time as new models, policies, and designs are introduced. For example, for a vehicle as an example of an object, the model and design of the vehicle may change over time as new models are introduced and old vehicles are retired. Vehicles may be modified, damaged, and may appear different from the showroom model due to loading or adaptation to weather conditions. Therefore, it is difficult to cover all vehicles in a video analytics system.

[0003] In the real world, a monitored object may be blocked by other objects. The mounting position of a camera capturing an image of a monitored object may change depending on the location, and the posture of the object in the image may change depending on the mounting position or mounting state of the camera. In such cases, it is often impossible to use a template for matching objects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,176,405 [Patent Document 2] U.S. Pat. No. 9,438,878 [Patent Document 3] U.S. Pat. No. 9,609,307 [Patent Document 4] U.S. Pat. No. 1,078,9717 [Patent Document 5] U.S. Pat. No. 9,613,137 [Patent Document 6] U.S. Patent No. 10007020 [Patent Document 7] Chinese Patent No. 104149716 [Patent Document 8] European Patent No. 2818340 [Patent Document 9] US Patent Application Publication No. 2019 / 0095877 [Patent Document 10] U.S. Patent No. 7,881,865 [Patent Document 11] International Publication No. 2020 / 262520 [Patent Document 12] U.S. Pat. No. 9,076,276 [Patent Document 13] U.S. Patent No. 9,218,698 [Non-patent literature]

[0005] [Non-Patent Document 1] 'Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision'. Niemeyer et. al CVPR 2020: 3501-3512 Summary of the Invention [Problem to be solved by the invention]

[0006] For example, in Non-Patent Document 1, a method is proposed to implicitly learn shape and texture representations. The method proposes to cast a ray along a direction w through a point u. A neural network is used to determine the occupancy probability, determine where the object boundary starts, and call points whose probability is greater than a set threshold τ. The neural network also predicts red, green, and blue color intensity values, represented by c=t(). The occupancy probability and texture are predicted for each point u in the image mask. However, this method assumes that the camera parameters are known and requires both extrinsic and intrinsic parameters to generate the output. The occupancy probability and texture values ​​are conditioned on global features generated by the neural network and do not take into account the depth and texture of nearby pixels. This causes a smoothing of the depth map, making the texture appear to be of uniform intensity.

[0007] Techniques for investigating object re-identification using 3D models and multiple views are disclosed as shown below.

[0008] In Patent Document 1, a method for vehicle re-identification using a neural network is proposed. In this method, first, a viewpoint is identified using a neural network. Then, an attention map is used to build a multi-view representation. This multi-view representation is then compared with an existing multi-view representation to re-identify the vehicle. Direct classification of viewpoints from images requires model training with different camera parameters. Furthermore, the inferred viewpoints may differ from the viewpoints already captured. Higher resolution in viewpoint classification and attention-based matching of intersections requires more data samples from many viewpoints. Multiple viewpoints may not be available at the time of inference.

[0009] Patent document 2 proposes a method for obtaining a 3D model of a 2D object in a scene. In this method, for each frame, the position and orientation of the object in the frame are determined. A depth map is generated and this map is used to generate a stereo image. It is assumed that a 3D model is available for the object under observation or can be obtained from a scanner. This assumption is not always possible. Furthermore, this method does not address the difficulty of knowing the camera parameters.

[0010] Patent Document 3 proposes a method for converting 2D video to 3D video where a 3D model is generated using machine learning. This method does not estimate camera parameters, but uses the generated 3D model to construct a stereoscopic image. Since the viewpoint of the image is not fixed, using a neural network to directly infer a depth map without providing camera parameters may cause the model to be overfitted, producing inaccurate mesh estimation and matching results.

[0011] US Patent No. 5,399,633 proposes a technique for learning the pose of moving objects by learning the correspondence of 3D feature points on a mesh to feature points found in 2D images. The approach discloses fitting camera parameters, but the approach relies on the presence and matching of feature points in an existing mesh model with feature points in the images. This approach increases feature matching errors if the mesh of the object being captured is not present in the training set. These errors in feature matching due to different appearances and shapes can result in an uneven 3D model.

[0012] Patent Document 5 proposes a method for identifying vehicle status using tags, using a system for checking vehicle status (ownership), which can identify the location of a stolen vehicle. However, tag-based identification requires the installation of sensors and tags on all vehicles.

[0013] US Patent No. 6,399,633 proposes a method for vehicle inspection using X-ray based images, in which images are compared with a valid state reference.

[0014] Patent Document 7 proposes a remote vehicle diagnosis system for diagnosing a fault using a sensor mounted on a vehicle.

[0015] A control and rescue system for wheel diameter change and high-speed tire burst is proposed in Patent Document 8. This system uses sensors mounted on the vehicle, which increases the deployment cost of the entire system.

[0016] Patent Document 9 proposes an image recognition system for vehicle damage detection. This system requires the construction of a training set with images of damaged and undamaged images. The use of the camera configuration used to collect the data may bias the system, limiting its applicability in uncontrolled configuration environments.

[0017] Patent Document 10 proposes a collision determination system that requires vehicle trajectory and behavior to perform collision detection prediction. Such trajectory may not always be available when an application monitors access to a location or when a continuous line of sight view is not available.

[0018] Patent Document 11 proposes a road management device to detect whether a vehicle is suitable to enter an automated driving lane based on a specified width limit. In practice, this requires that the detection device be trained to detect the width at a specific viewpoint.

[0019] Patent Document 12 proposes a technology that uses vehicle identifier information such as auction records, accident records, and vehicle numbers to alert users to selected events.

[0020] Patent Document 13 discloses a technique that uses vehicle sensor data for damage detection, but this technique has limited applicability to operators who do not have access to such data.

[0021] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique that can easily and appropriately evaluate the state of an object. [Means for solving the problem]

[0022] In order to achieve the above-mentioned object, an object state evaluation system according to one aspect is an object state evaluation system that evaluates the state of an object in an image, and has a processor, wherein the processor acquires an image including an object to be evaluated, generates an evaluation image in which the object in the image is in a first posture, and judges whether or not the object is in the specified state based on the evaluation image and a judgment template image in which the object is in a specified state and in the first posture. Effect of the Invention

[0023] According to the present invention, the state of an object can be easily and appropriately evaluated. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating processing in an information processing system according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating details of the processing of the object state evaluation system according to one embodiment. [Figure 4] FIG. 4 is a diagram illustrating a part of the processing of an object state evaluation system according to an embodiment. [Diagram 5] FIG. 5 is a diagram illustrating the process of the template generator according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a process of the template generator according to an embodiment. [Figure 7] FIG. 7 is a diagram illustrating another example of the process of the template generator according to an embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a generative model according to an embodiment. [Figure 9] FIG. 9 is a diagram illustrating a sampling process in a generative model according to one embodiment. [Figure 10] FIG. 10 is a diagram illustrating the processing of a camera calibration system according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating a portion of a mesh and texture prediction system according to one embodiment. [Figure 12] FIG. 12 is a diagram illustrating another portion of a mesh and texture prediction system according to an embodiment. [Figure 13] FIG. 13 is a diagram for explaining a texture generation process of the mesh and texture prediction system according to the modified example. [Figure 14] FIG. 14 is a diagram showing a GUI screen of an object state evaluation system according to an embodiment. [Figure 15] FIG. 15 is a block diagram of an object tracking system for tracking an object including an object state assessment system according to one embodiment. [Figure 16] FIG. 16 is a configuration diagram of an infrastructure management system for managing an infrastructure including an object state evaluation system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0026] Fig. 1 is a diagram showing the overall configuration of an information processing system according to an embodiment of the present invention, and Fig. 2 is a diagram explaining the processing in the information processing system according to an embodiment of the present invention.

[0027] The information processing system 1 includes an image acquisition system 2, an object state evaluation system 10, and a display system 3.

[0028] The image acquisition system 2 is a system that acquires images of the infrastructure (core facilities) to be managed or the area to be monitored. The image acquisition system 2 includes a camera that captures images, such as an RGB camera and / or a thermal camera, and a transmitter that transmits images (frame images) acquired by the camera to the image acquisition system 2. The camera is placed in close proximity to the infrastructure to be managed or the area to be monitored.

[0029] The object state evaluation system 10 is configured with, for example, one or more computers having a processor and a memory. In this embodiment, the object state evaluation system 10 is configured with, for example, one or more servers on a cloud.

[0030] The object state assessment system 10 includes an object detection and segmentation system 11, a camera calibration system 12, an object tracking system 13, an image rendering system 14, an image matching system 15, a database 16, and a template generator 18. Note that the template generator 18 is not necessarily required in the object state assessment system 10. The object state assessment system 10 is activated only when the image acquisition system 2 detects the presence of an object to be monitored, and may be powered off in other cases (such as when processing is completed).

[0031] The object detection and segmentation system 11 detects an object to be evaluated in an image acquired from the image acquisition system 2, and performs processing to segment the object. The object detection and segmentation system 11 estimates from the image a class (specific information) representing the attributes of the object, an object score, coordinates at which the object is detected (detection coordinates), a segmentation mask for segmenting the object, pose keypoints, and depth. The class represents the type of object, for example, if the object is a car, the make, model, and color of the car. The score represents the confidence that the object is of the predicted class. The detection coordinates represent the polygonal region boundary of the object. The segmentation mask represents a pixel-by-pixel classification of the presence / absence of an object instance. The depth represents the distance (depth) from the camera for each pixel of the detected object instance. The pose keypoints represent the pixel coordinates of the part that indicates the pose of the object.

[0032] The object detection and segmentation system 11 may be, for example, a neural network model similar to the model described in Mask-RCNN (Mask R-CNN. Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross B. Girshick IEEE Trans. Pattern Anal. Mach. Intell. 42(2): 386-397 (2020) Internet: URL=https: / / dblp.org / db / journals / pami / pami42.html#HeGDG20) with the addition of an additional head to predict depth and object pose in addition to detection coordinates, scores, classes, and segmentation masks.

[0033] The camera calibration system 12 uses the pose keypoints of the detected objects to generate a camera matrix that transforms three-dimensional coordinates (3D coordinates) into coordinates in an image, which is used to generate a 3D representation of the detected objects.

[0034] The object tracking system 13 tracks the detected objects using the scores, segmentation masks, detection coordinates, and classes output from the object detection and segmentation system 11. The object tracking system 13 assigns a track ID for uniquely identifying every object instance detected from a frame image, and outputs the track ID and the class of the object.

[0035] The image rendering system 14 includes a mesh and texture prediction system 21, a rendering system 22, and a refinement system 23. Based on the track ID and the class, the image rendering system 14 determines whether a 3D model of the corresponding object is available.

[0036] The mesh and texture prediction system 21 predicts the 3D mesh and texture of an object to generate a 3D model if a corresponding 3D model of the object is not available. The mesh and texture prediction system 21 may use the camera matrices generated by the camera calibration system 12 to more accurately generate the 3D mesh of the object.

[0037] The rendering system 22 generates a synthetic image (rendered image: evaluation image) of the object in a given pose using a corresponding 3D model of the object.

[0038] The refinement system 23 refines or stylizes the composite image generated by the rendering system 22 and outputs it. Note that the refinement system 23 may pass the composite image generated by the rendering system 22 to the next process as is.

[0039] Various types of information are stored in the database 16. In this embodiment, the database 16 stores templates 17 (template images) for matching with an image of an object.

[0040] The template generator 18 creates templates of predefined states and predefined poses for matching with images of the object of interest.

[0041] The image matching system 15 uses feature similarities between the template created by the template generator 18 or the templates in the database 16 and the image of the target object to estimate matching attributes such as color, car make, model, appearance similarity, etc. between the object in the template and the object in the image, and outputs an evaluation result including matching or non-matching features and similarity to the template. Note that in the case of re-identifying a particular object, the template may be a previously captured image of the object.

[0042] The display system 3 displays (outputs) various information based on the evaluation results (matching attributes) of the states of objects in the images by the object state evaluation system 10. The display system 3 may display the evaluation results, or may display an alert in response to the evaluation results. The output of the display system 3 may be used as an input to a cloud application 24, or may be stored in a database 16 for later use.

[0043] FIG. 3 is a diagram illustrating details of the processing of the object state evaluation system according to one embodiment.

[0044] The object detection and segmentation system 11 detects an object from an input image 30, identifies and outputs the object's class and orientation, and also outputs a segmented image 31 obtained by cutting out a partial image including the object from the image 30.

[0045] The image rendering system 14 determines, based on the class, whether a 3D model of the corresponding object is available (S11).

[0046] As a result, if a 3D model of the corresponding object is available (S11: Yes), the rendering system 22 uses the available 3D model to generate a synthetic image of the object in a predetermined pose (e.g., the pose assumed in the template: the first pose).

[0047] On the other hand, if a 3D model of the corresponding object is not available (S11: No), the mesh and texture prediction system 21 predicts the 3D mesh and texture of the object using the pose, segmented image, etc., and generates a 3D model of the object based on the predicted 3D mesh and texture. In this case, the rendering system 22 generates a synthetic image of the object in a predetermined pose (first pose) using the 3D model generated by the mesh and texture prediction system 21.

[0048] The generative model 33, which is an example of the refinement system 23, performs a process of refining the composite image generated by the rendering system 22. The generative model 33 converts the style of the composite image to a predetermined style (a style specified by the user or a default style). The generative model 33 may also be used to suppress or emphasize details in the image. The generative model 33 may also perform processing according to the class of the object. The generative model 33 may be, for example, a text-conditional diffusion model described in the literature (Null-text Inversion for Editing Real Images using Guided Diffusion Models. CoRR abs / 2211.09794, Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, Daniel Cohen-Or, (URL=https: / / dblp.org / db / journals / corr / corr2211.html#abs-2211-09794)).

[0049] A feature extractor (feature extraction model) 35 of the image matching system 15 receives as input a composite image generated by the rendering system 22 or a composite image converted by the generative model 33 (S12) and extracts features of the composite image, and also extracts features of a template stored in the database 16 or a template created by the template generator 18. The image matching system 15 compares the features of the composite image and the template in Euclidean space and determines whether the distance in the Euclidean space is equal to or less than a threshold (S13), and if it is equal to or less than the threshold (S13: Yes), determines that they are the same, and if it is not equal to or less than the threshold (S13: No), determines that they are different.

[0050] FIG. 4 is a diagram illustrating a specific example of a part of the processing of the object state evaluation system according to an embodiment.

[0051] For example, when image 41 is of a pickup truck, if a 3D model of the pickup truck is not available (S11: No), mesh and texture prediction system 21 predicts mesh 42 and texture 43 using the pose, image 41, etc. Next, rendering system 22 creates a 3D model using generated mesh 42 and texture 43 as input, generates image 44 of the pickup truck in a desired pose using the 3D model, and applies texture 43 to generate image 45 of the pickup truck in a predetermined pose with the texture reflected.

[0052] FIG. 5 is a diagram illustrating the process of the template generator according to an embodiment.

[0053] The template generator 18 includes a local rule store 51, a cloud-based rule processor 52, a text rule generator 53, a 3D model generator 54, and a generative model 55.

[0054] The local rule storage unit 51 stores rules and regulations (geographical rules, area management rules, time- and season-dependent rules, etc.) in the area monitored by the camera of the image acquisition system 2, and information related to these.

[0055] The cloud-based rule processor 52 analyzes the information in the local rule storage unit 51 and obtains information related to restrictions on objects in the area to be monitored (restriction information).

[0056] The text rule generator 53 generates text rules (text information) that represent conditions that satisfy the constraints on the object based on the information obtained by the cloud-based rule processor 52 .

[0057] The template generator 18 determines whether a 3D model of an object corresponding to the text rule is available (S51).

[0058] If a 3D model is available (S51: Yes), the 3D model generator 54 uses the 3D model to generate a template (judgment template image) corresponding to the object in a predetermined pose.

[0059] On the other hand, if a 3D model is not available (S51: No), the generative model 55 inputs the text rule and generates a template (judgment template image) corresponding to an object in a predetermined pose that corresponds to the text rule.

[0060] Next, a specific example of the process of template generator 18 will be described.

[0061] FIG. 6 is a diagram illustrating an example of a process of the template generator according to an embodiment.

[0062] Here, the local rule storage unit 51 stores information that snow is present in the area to be monitored and that snow chains are required for vehicles in this area. Note that, as specific information, for example, the information that snow is present in the area to be monitored and the information that snow chains are required for vehicles in the case of snow may be managed separately.

[0063] The cloud-based rule processor 52 analyzes the currently restricted rules in the monitored area from the information in the local rule memory unit 51. Specifically, the cloud-based rule processor 52 analyzes that there is snow in the monitored area and that the use of snow chains is required for vehicles in this area, based on the fact that there is snow in the monitored area and that the use of snow chains is required for vehicles in this area.

[0064] The text rule generator 53 creates text rules that mean attributes required for an object from the analysis information by the cloud-based rule processor 52. For example, the format of the text rule is <vehicle type>+with+<attribute>. When the vehicle type of the object is a pickup truck, the text rule generator 53 generates the text rule 61 of "Pick up truck with chains on tyres."

[0065] The generative model 55 receives a conditioned image (reference template image) 62 and a text rule 61 as input, and generates a template 63 that conforms to the text rule 61, that is, a template that is a pickup truck and uses chains.

[0066] FIG. 7 is a diagram illustrating another example of the process of the template generator according to an embodiment.

[0067] Here, the local rule storage unit 51 stores information that the monitoring target is a forest area and that it is desired to detect illegal loading of cut trees.

[0068] The cloud-based rule processor 52 analyzes the rules currently restricting the area to be monitored from the information in the local rule memory unit 51. Specifically, the cloud-based rule processor 52 analyzes that the area to be monitored is a forest area and is not illegally loading cut trees.

[0069] The text rule generator 53 creates a text rule that means an attribute required for an object from the analysis information by the cloud-based rule processor 52. For example, the format of the text rule is <vehicle type>+with+<attribute>. When the vehicle type of the object is a pickup truck, the text rule generator 53 generates the text rule 71 of "Pick up truck with loaded logs."

[0070] The generative model 55 inputs the conditioned image 62 and the text rule 71 and generates a template 73 that conforms to the text rule 71, that is, a template that is a pickup truck and is suitable for loading felled trees.

[0071] FIG. 8 is a diagram illustrating an example of a generative model according to an embodiment.

[0072] The generative model 55 includes an encoder 131, a forward diffusion unit 132, a neural network 133, a neural network 134, an encoder-decoder model group 135, and a decoder 136. The encoder 131 encodes an image X into a feature Z. The forward diffusion unit 132 adds T steps of noise to the feature Z, and obtains Z. T Generate.

[0073] Neural network 133 takes the tokenized text description as input and generates text embeddings. Neural network 134 tokenizes the null text and generates null text embeddings.

[0074] The encoder-decoder model group 135 includes encoder-decoder models 135A and 135B, which are the same model. The encoder-decoder model 135A inputs the features Zt, the text embedding, and the time step, and denoises the embedding. The encoder-decoder model 135B inputs the features Zt, the null text embedding, and the time step, and denoises the embedding. The processing of the encoder-decoder model 135A and the encoder-decoder model 135B can be performed in parallel or in series for multiple time steps. The embeddings generated in the final step are reweighted and combined in the latent space and input to the decoder 136. The decoder 136 generates a synthetic image x̂ using the combined embeddings.

[0075] FIG. 9 is a diagram illustrating a sampling process in a generative model according to one embodiment.

[0076] Figure 9 shows a graphical model for the inverse sampling process of the diffusion-based generative model 55. This inverse sampling process is similar to the inverse diffusion process described in the literature (Null-text Inversion for Editing Real Images using Guided Diffusion Models. Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, Daniel Cohen-Or:CoRR abs / 2211.09794 (2022)(URL=https: / / dblp.org / db / journals / corr / corr2211.html#abs-2211-09794)).

[0077] The sampling steps are represented by index T. Index 0 indicates the denoised sample and index T represents the sample with additive noise after T steps.

[0078] Using standard diffusion inversion processing, Z *Using the approach presented in the paper, an embedding vector is generated, which is represented by the variable Z - t In this embodiment, the embedded trajectory Z -- t (In the diagram, a double - above the Z) is generated.

[0079] In the null text inversion, we try to keep the original diffusion trajectory Z* close to the original, but in this embodiment, we relax the requirement for a trade-off where the region specified by the mask has a low reconstruction loss. The loss is expressed as shown in Equation (1).

[0080]

number

[0081] Here, Φ t indicates padding with null text = "", Z t Let denote the encoding of the noisy sample in at step t when the trajectory of the denoising diffusion implicit model (DDIM) is followed, and Z -- t Let denote the noise-encoded samples when the above minimization is performed on the parameters of the network that generates the null embedding, C denotes the embedding of the given text used for conditioning, and f γ denotes the classification model with parameters γ, p denotes the output of the decoder model, and I (mask) denotes the user-defined mask image to be edited, and I (initial) indicates an image to be edited, α indicates a constant parameter, and β indicates a constant parameter. The neural network 133 and the neural network 134 in Fig. 8 are trained using the loss function of equation (1).

[0082] FIG. 10 is a diagram illustrating the processing of a camera calibration system according to an embodiment.

[0083] The camera calibration system 12 estimates a camera matrix that maps 3D coordinates in the world coordinate system to 2D coordinates in the pinhole camera camera image 81. In the world coordinate system, parallel lines do not intersect, but in the camera image coordinate system, parallel lines converge to a point (the vanishing point). The camera calibration system 12 selects keypoints from the image that correspond to parts of the object that are known to be parallel (e.g., corners of the top loop of a car), and recognizes the vanishing point based on pairs of parallel lines based on these. The camera calibration system 12 calibrates the camera based on the recognized vanishing point.

[0084] A method of calibration using vanishing points is described in, for example, the literature (Camera calibration with two or three vanishing points: Radu Orghidan, Joaquim Salvi, Mihaela Gordan, Bogdan Orza: FedCSIS 2012: 123-130 (URL=https: / / annals-csis.org / proceedings / 2012 / pliks / 110.pdf)) and the literature (Using vanishing points for camera calibration. Int. J. Comput. Vis. 4(2): 127-139 (1990)(URL=https: / / dblp.org / db / journals / ijcv / ijcv4.html#CaprileT90)).

[0085] FIG. 11 is a diagram illustrating a portion of a mesh and texture prediction system according to one embodiment.

[0086] In the literature (Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision. Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, Andreas Geiger:CVPR 2020: 3501-3512, (URL=https: / / dblp.org / db / conf / cvpr / cvpr2020.html#NiemeyerMOG20)), a convolutional neural network is used to generate a global context embedding, whereas the mesh and texture prediction system 21 of the present embodiment has an attention-based mechanism for detecting an embedding vector for a patch in the image 91.

[0087] The mesh and texture prediction system 21 includes a detector 92, a linear project matrix 96, and a context embedding generator 98. The detector 92 is, for example, a neural network model, and detects the class of an object instance from the image 91.

[0088] A set of patches 94 into which the image 91 is divided, and position embeddings 95 corresponding to the patches are input to the linear project matrix 96. The linear project matrix 96 outputs the features of each patch (patch features 97).

[0089] The context embedding generator 98 includes a scaled dot-product attention 99, a standardization unit 100, and an MLP (multi-layer perceptron) 101, and generates a context embedding Z using the class embedding 93 detected by the detector 92 and the patch features 97 as input.

[0090] FIG. 12 is a diagram illustrating another portion of a mesh and texture prediction system according to an embodiment.

[0091] The mesh and texture prediction system 21 further includes a ray casting unit 111 and a depth and texture prediction model 112 .

[0092] A ray casting unit 111 inputs the camera matrix and the image 110, projects light from pixel u of the image, and calculates world coordinates that exist in the volumetric region of the object along the light ray.

[0093] The depth and texture prediction model 112 uses the world coordinates calculated by the ray casting unit 111 and the context embedding generated by the context embedding generation unit 98 to predict the color and probability for coordinates within the instance, and outputs a tuple (d(depth), R(red), G(green), B(blue)) for each input pixel of the instance.

[0094] FIG. 13 is a diagram for explaining a texture generation process of the mesh and texture prediction system according to the modified example.

[0095] In the above embodiment, the texture is predicted by the configuration shown in FIG. 12, but the texture may be predicted by a texture generation model 121 shown in FIG.

[0096] The texture generation model 121 may be a multi-layer neural network model and may include multi-head attention. The texture generation model 121 takes as input a camera matrix, a mesh, and images of multiple views to predict RGB or intensity values ​​for multiple pixel locations in the texture field 122.

[0097] FIG. 14 is a diagram showing a GUI screen of an object state evaluation system according to an embodiment.

[0098] The GUI screen 200 includes a device selection area 201, a mode selection area 202, a rule update selection area 203, a matching method selection area 204, an image selection area 205, a text prompt 206, a custom rule selection area 207, an inferred 3D model display area 208, an edit 3D model button 209, a custom refinement selection area 210, a location selection area 211, an alert display area 212, an attribute display area 213, and a top match display area 214.

[0099] The device selection area 201 is an area for selecting the image acquisition system 2 to be analyzed. The mode selection area 202 is an area for selecting an application case such as access management, tracking, search, etc. Based on the case selected or allowed in this area, the display components of the GUI screen 200 are made active or inactive. Also, based on the case, the system processing may be modified by adding or bypassing the processing flow. The system processing controls how the output is visualized and used, and is used to open other graphical interfaces.

[0100] The rule update selection area 203 is an area for selecting the source of text description for template generation in the template generator 18, image synthesis in the refinement system 23, etc. The matching method selection area 204 is an area for selecting the feature similarity and changing the behavior of the image matching system 15. The image selection area 205 is an area for selecting a conditional image when synthesizing a template. The text prompt 206 is an area for inputting a text prompt for the conditional synthesis of the template image.

[0101] The custom rule selection area 207 is for generating graphical node based text. It allows the user to create scripts that generate text descriptions of conditional synthesis using connections of graphical nodes. The inferred 3D model display area 208 displays the generated 3D model. The edit 3D model button 209 is for editing and regenerating the synthesized 3D model. It can also be used to provide an external 3D model or edit an existing or synthesized model. The custom refinement selection area 210 is for selecting a refinement strategy for the rendered image in the refinement system 23. The location selection area 211 is for selecting the image acquisition location where the analysis is performed.

[0102] The alarm display area 212 is an area where alarms detected in the display system 3 are displayed. The attribute display area 213 is an area where the unique ID and attributes of the analyzed object are displayed. The top match display area 214 is an area where the top matching objects among the searched objects and their scores are displayed.

[0103] FIG. 15 is a block diagram of an object tracking system for tracking an object including an object state assessment system according to one embodiment.

[0104] The object tracking system 300 is a system for tracking objects, and includes monitoring devices 301 (301A, 301B, 301C, etc.) installed at multiple locations, a receiver 304, a database (DB) 5, an alert generation system 4, and an object state evaluation system 10.

[0105] The monitoring device 301 includes cameras 303 (303A, 303B, 303C) that capture images of a predetermined area where the device is installed, and transmitters 302 (302A, 302B, 303C) that transmit images captured by the cameras 303.

[0106] The receiver 304 receives the image of the object transmitted from the transmitter 302 , stores the image in the DB 5 , and passes it to the object state assessment system 10 .

[0107] The object state evaluation system 10 tracks the position of the target object by matching the image of the object acquired from the receiver 304 or DB 5, and notifies the information to the alert generation system 4. The alert generation system 4 displays an alert indicating the position of the target object on a specified display device that is the notification destination.

[0108] According to the object tracking system 300, even if the pose of an object photographed by the camera 303 of a different monitoring device 301 is different, the object state evaluation system 10 generates a composite image of the object in a pose corresponding to the template. This allows appropriate object matching and appropriate tracking of the object's position.

[0109] Fig. 16 is a configuration diagram of an infrastructure management system for managing an infrastructure including an object inter-state system according to an embodiment. The infrastructure management system 400 in Fig. 16 is a system for managing whether a vehicle, which is an example of an object, can access (enter) a tunnel, which is an example of an infrastructure.

[0110] The core facility management system 400 includes a receiver 402 installed in a core facility 401, an alarm device 403, a monitoring device 404 arranged to monitor the route to the core facility 401, a receiver 407, a database (DB) 5, an alert generation system 6, and an object state evaluation system 10.

[0111] The surveillance device 404 includes a camera 406 that images a predetermined area of ​​the route to the infrastructure, and a transmitter 405 that transmits the images captured by the camera 406 .

[0112] The receiver 407 receives the image of the object transmitted from the transmitter 405 , stores the image in the DB 5 , and passes it to the object state evaluation system 10 .

[0113] The object state evaluation system 10 identifies an object from an image of the object acquired from the receiver 407 or DB 5, and creates a template for the object in a state where the object can access the infrastructure 401 (here, at a height where the object can enter the tunnel). The object state evaluation system 10 evaluates whether the object can access the infrastructure 401 by matching the template with the image of the object, and notifies the alert generation system 6 of the result. When the alert generation system 6 receives a notification that the object cannot access the infrastructure 401, the alert generation system 6 causes the alarm device 403 to display an alert so that the object can confirm the alert, or generates an alarm sound, via the receiver 402. This allows the user of the object to properly understand that the infrastructure 401 cannot be accessed.

[0114] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.

[0115] For example, in the above embodiment, some or all of the processing performed by the processor of the object state evaluation system 10 executing the program may be performed by a hardware circuit. Also, the program in the above embodiment may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium).

[0116] For example, in the above embodiment, object attributes are matched using RGB images or thermal images, but information detected by, for example, an acoustic sensor, a gas sensor, an explosive sensor, a pressure sensor, a piezoelectric transducer, etc. may also be used. [Explanation of symbols]

[0117] 1...information processing system, 2...image acquisition system, 3...display system, 10...object state assessment system, 11...camera calibration system, 12...object detection and segmentation system, 13...object tracking system, 14...image rendering system, 15...matching system, 16...database, 17...template, 18...template generator.

Claims

1. 1. An object state assessment system for assessing a state of an object in an image, comprising: A processor is included. The processor, Obtaining an image including an object to be evaluated; generating an evaluation image in which the object in the image is in a first pose; Whether or not the object is in a predetermined state is determined based on the evaluation image and a determination template image in which the object is in a predetermined state and in the first orientation. Object condition assessment system.

2. The method further includes a storage unit for storing restriction information relating to restrictions on the object, The processor, generating text information indicating an object in a state that conforms to the restriction indicated by the restriction information based on the restriction information; A template image corresponding to the object in a state that satisfies the restriction is generated as the judgment template image based on the text information and a reference template image of the object. The object state assessment system of claim 1 .

3. The processor, The judgment template image is generated using a neural network. The object state assessment system of claim 2 .

4. The processor, If the object is not in the predetermined state, an alert is sent to a predetermined notification destination. The object state assessment system of claim 1 .

5. The processor, Extracting specific information that identifies the object based on the image; determining whether or not a 3D model of an object corresponding to the specific information exists, and if a 3D model of the object exists, generating the evaluation image in which the object is in the first pose based on the 3D model; The object state assessment system of claim 1 .

6. The processor, When a 3D model of the object corresponding to the specific information does not exist, a 3D mesh and a texture of the object are predicted based on a partial image of the object of the image, a 3D model is created based on the 3D mesh and the texture, and the evaluation image in which the object is in the first posture is generated based on the created 3D model. The object state assessment system of claim 5 .

7. The processor, Based on the image, coordinates of a plurality of points of the object in the image are identified, depths of the plurality of points are estimated, a posture of the object is identified based on the depths of the plurality of points, and an evaluation image having the first posture is generated based on the identified posture. The object state assessment system of claim 1 .

8. The processor, Identifying a vanishing point for the image based on the image; and calculating a camera matrix for the image based on the vanishing point; Using the camera matrix, estimate the depth of a number of points of an object in the image. The object state assessment system of claim 7.

9. The processor, Using the camera matrix, the images of the multiple views of the object, and the mesh of the object, generate a texture for the object in the image. The object state assessment system of claim 8.

10. The object state evaluation system includes: It is activated when a certain condition is detected, processed, and then powered off. The object state assessment system of claim 1 .

11. An object state evaluation method by an object state evaluation system for evaluating a state of an object in an image, comprising: The object state evaluation system includes: Obtaining an image including an object to be evaluated; generating an evaluation image in which the object in the image is in a first pose; Whether or not the object is in a predetermined state is determined based on the evaluation image and a determination template image in which the object is in a predetermined state and in the first orientation. Object condition assessment methods.

12. An object state evaluation program for executing on a computer constituting an object state evaluation system for evaluating a state of an object in an image, The computer includes: Acquiring an image including an object to be evaluated; generating an evaluation image in which the object in the image is in a first pose; determining whether or not the object is in a predetermined state based on the evaluation image and a determination template image in which the object is in a predetermined state and in the first orientation; Object condition assessment program.

Citation Information

Patent Citations

  • Target object generation method and device, electronic equipment and storage medium

    CN113590858A

  • Image editing model training method and device, image editing method and device, electronic apparatus, storage medium and computer program

    JP2022172173A

  • Damage detection from multi-view visual data

    JP2022523478A

  • Online vehicle maintenance

    CN104149716A

  • Safety driving control and rescue system for wheel diameter change and high-speed tire burst

    EP2818340A1

Cited By

  • Evaluation device and evaluation method

    JP7931915B1