Method for discovering system error of image-based object detector
By generating object images with different visual attributes and poses through a multi-step generation pipeline and a generative machine learning system, the problem of difficult identification of systematic errors in object detectors is solved, and unsupervised error assessment and performance improvement are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2026-04-17
AI Technical Summary
Existing image-based object detectors are prone to systematic errors in robotic systems, and these errors are difficult to identify and assess effectively using traditional methods, especially in robot motion planning where they may lead to potentially dangerous behaviors.
By using a multi-step generation pipeline, we can generate object images with different visual attributes and poses using a generative machine learning system, place them in a background scene, evaluate the performance of the object detector, and identify and assess system errors.
It realizes the evaluation of system errors of object detectors in an unsupervised manner, provides more refined object pose control, can identify and mitigate system errors, and improves the reliability of object detectors and the reliability of performance evaluation.
Smart Images
Figure CN121889835A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a computer implementation method for detecting systematic errors in an image-based object detector, a method for training an object detector, a computer program, and a machine-readable storage device.
[0002] Existing technology Zhang et al., “Adding Conditional Control to Text-to-Image Diffusion Models, February 10, 2023, https: / / arxiv.org / abs / 2302.05543, disclosed the ControlNet neural network for image generation.”
[0003] Kirillov et al., “Segment Anything”, April 5, 2023, https: / / arxiv.org / abs / 2304.02643, disclosed an algorithm for segmenting anything for image segmentation.
[0004] Liu et al., “Zero-1-to-3: Zero-shot One Image to 3D Object”, March 20, 2023, https: / / arxiv.org / abs / 2303.11328, disclosed a method to change the camera viewpoint of an object given a single RGB image.
[0005] The stable diffusion fix is publicly available in the online resource https: / / huggingface.co / runwayml / stable-diffusion-inpainting. Background Technology
[0006] Image-based object detectors are a key building block for a wide variety of applications. Particularly in robotics, such as autonomous vehicles, these object detectors are crucial for inferring the virtual environment, which is then used to plan the robot's movements. Because robot movements can only be planned within a virtual environment, errors from the object detector can propagate throughout the system, leading to potentially dangerous behaviors.
[0007] Most object detectors are based on machine learning systems. Due to the arbitrary uncertainties in the input images, random errors will occur with a non-zero probability. However, object detectors can also exhibit systematic errors. For example, an object detector may consistently perform poorly for objects photographed from a particular angle compared to its average performance.
[0008] Identifying such systematic errors is highly desirable because they are not related to random effects in the object detector's input, but rather to systematic defects. Typically, such defects can be mitigated by training the object detector on more data from classes exhibiting systematic errors (in the example above, more images of objects taken from a specific angle or with angle variations around that angle).
[0009] However, discovering such systematic errors is a critical task because they can typically only be revealed based on a large number of test images. Manually annotating these test images is impractical, as the root of systematic errors can lie in a wide variety of object attributes (e.g., color, texture, reflectivity, size, position, rotation). Furthermore, the inventors found that when using generative methods to simply determine test images based on textual cues, they suffer from insufficient attribute binding. That is, generative methods often misinterpret which object should acquire the attributes expected in the textual cues. Moreover, fine-grained control over object pose is unreliable, and confusion frequently occurs even at the level of "back view" versus "side view" of the object. This limited control over object pose makes evaluating object detectors using such generative methods infeasible, as there is no reliable process to control the position of objects in the image and what attributes they actually possess.
[0010] Advantageously, the method having the features of independent claim 1 allows for greater control over the appearance and pose of individual objects by replacing a single, monolithic text-to-image generative model with a multi-step generation pipeline.
[0011] Finer control over object pose allows this invention to be applied to identify systematic errors in object detectors. For example, by synthesizing different types of vehicles (semantic dimension) and altering their orientation, object detectors that cannot reliably detect rear views of certain vehicle types (such as pickup trucks) can be identified. By also controlling the location of objects in the image, the system provides ground-truth information to the object detector in terms of both semantics (class) and bounding boxes.
[0012] This allows for the evaluation of the effects of individual attributes and combinations of attributes in an unsupervised manner, meaning the method can be readily provided with large numbers of unlabeled images and still offer a meaningful assessment of the systematic error of the object detector. Summary of the Invention
[0013] In a first aspect, the present invention relates to a computer implementation method for determining the systematic error of an image-based object detector, comprising the following steps: • Determine the first image of an object from multiple different types of objects; • Determine the edges or pose of an object; • Determine at least one visual attribute from a set of possible visual attributes of the object to be tested for systematic error; • A second image is generated by a generative machine learning system using at least one visual attribute as a cue or for the generative machine learning system, thereby generating an image of an object displaying at least one visual attribute, wherein the generative machine learning system is additionally conditioned on the determined edges or pose of the object. • The object from the second image is placed into the image of the background scene to generate the third image; • Evaluate the object detector on the third image and use the evaluation to determine the systematic error of the object detector.
[0014] In summary, this method can be understood as determining a first image of the object to be evaluated, extracting edges from the first image, selecting (or determining) at least one, but preferably multiple, attributes that the object should exhibit (e.g., a different color or texture than the object currently has), using these attributes as desired properties and using the object's edges as hints to generate an image, placing the object in front of the background, and then evaluating the object detector on the generated image.
[0015] Since at least one visual attribute and location of the object are known, the baseline ground truth annotation (e.g., bounding box) can be directly inferred and used as the expected result for the object detector. Specifically, the detection of other objects in the third image can be ignored, and only the object inserted in the third image can be considered.
[0016] An object detector can be understood as a machine learning system, particularly one that includes or consists of neural networks and preferred preprocessing and post-processing procedures (e.g., non-maximum suppression). An object detector can be configured to receive a third image as input and provide object detection as output, where object detection can be, in particular, in the form of bounding boxes, and preferably, the classes of objects enclosed by the bounding boxes. In other words, the object detector operates directly on the values of the image (pixels, voxels, depth, etc.).
[0017] Object detectors can be presented in a trained form. That is, an object detector can be the result of training an object detector on a suitable set of training data.
[0018] The performance of an object detector can be determined in particular by regard to misclassification (e.g., an object is correctly located but assigned the wrong class) or false negatives (e.g., the object detector does not detect an object inserted into a third image).
[0019] The third image can be understood as input to the object detector.
[0020] Typically, an image can be understood as a measurement from an optical sensor or the result of processing such a measurement. For example, an optical sensor may record pixel values of the environment, where processing may then include selecting only a subset of the measurements as an image and / or smoothing the measurements to determine an image and / or extracting information from the measurements, i.e., determining a depth map from the measurements and using that depth map as an image.
[0021] Suitable optical sensors include cameras, thermal cameras, lidar sensors, radar sensors, and ultrasonic sensors.
[0022] An image can also be composed of multiple different measurements, i.e., measurements recorded by a stereo camera. Here, the image consists of two "sub-images".
[0023] Images can typically be stored as a three-dimensional tensor, including width and height dimensions, along which discrete units of measurement (e.g., pixels) are positioned. The depth dimension can be adapted to the data being processed. For an RGB image, the depth dimension can consist of three elements from the red, green, and blue channels. For a stereo camera, the channels of a sub-image can be stacked along the depth dimension, where the depth dimension therefore comprises six elements.
[0024] Edge determination can be achieved using the Canny edge detector, in particular. To determine the pose of an object, known pose estimation methods can be used, especially those based on neural networks.
[0025] The edges or pose of the object can then be used as conditional input to guide the generative machine learning system. Based on this guidance, the system determines that the second image contains objects of the same or similar shape, even though they expose at least one different visual attribute.
[0026] Visual attributes can generally be understood as semantic concepts that characterize the appearance of an object. For example, visual attributes can characterize color, reflectivity, level of dirt, level of occlusion, clothing, stickers or other elements applied to or covering the object, the age of the object, and / or the material used to make the object or at least a part of the object.
[0027] This set of possible visual attributes characterizes the visual properties that will be examined to determine whether they contribute to the systematic error behavior of the object detector. They can be considered, in particular, at least as a part of the object detector's Operational Design Domain (ODD). This set of possible visual attributes can be determined by the user and thus supplied to the method.
[0028] This method can be run iteratively, wherein in each iteration, a visual attribute or a set of visual attributes is preferably randomly selected from a list of potential visual attributes.
[0029] Then, visual attributes are used in the input (i.e., the cue) to the generative machine learning system to determine the second image. Visual attributes are particularly useful in cuees that combine objects with visual attributes. For example, if the object is a car and the visual attribute is green, the cue could be "green car". Similarly, other visual attributes can be integrated into the cue so that the second image displays the object based on the determined visual attributes.
[0030] The generative machine learning system is preferably a ControlNet configured to accept edges or poses as conditional inputs. A first image can be provided as input to the generative machine learning system. Alternatively, the object can be "cut out" from the first image. This can be achieved by segmenting the object using a segmentation algorithm (e.g., segmenting everything) to determine the object's pixels. All other pixels can then be set to a uniform background, such as white.
[0031] Placing an object from a second image into a background image can be understood as copying the pixels representing the object in the second image to their positions in the background image. Pixel copying can be achieved by segmenting the object as described above and copying all pixels belonging to the segmented object into the background image. The background image can be specifically chosen as a predictable background for displaying the object detector's ODD. For example, the object detector can be used in robots such as autonomous vehicles, and the background image can be chosen to display urban, rural, or highway driving scenes.
[0032] Preferably, the object is placed in the background image using stable diffusion inpainting. Preferably, the stable diffusion inpainting model can fine-tune the image from the operational design domain of the object detector. Preferably, LoRA is used to achieve this fine-tuning.
[0033] The inventors discovered that using stable diffusion repair results in objects being embedded most realistically into the background image.
[0034] The object detector can then be evaluated with respect to objects in the third image. That is, when evaluating the performance of the object detector, other objects that already existed in the background image before the objects obtained from the generative machine learning system were inserted can be disregarded.
[0035] Evaluating object detectors can specifically include calculating performance values, also known as “determining the performance of the object detector.” Performance values can be determined based on performance metrics that measure the “goodness” of the object detector’s performance. Common object detection metrics can be used, such as mean precision, mean precision, AP50, AP75, mean recall, mean recall, accuracy, or false negative rate.
[0036] You can also inject multiple objects into the background image, that is, use the first four steps multiple times to identify different objects and place all objects into the background image.
[0037] Advantageously, this method allows for unsupervised testing of the object detector regarding visual attributes. That is, the method can be run on one or more images to determine the object detector's performance without requiring any annotations, as the annotations (i.e., the object's visual attributes, the object's location, and the object's class) are known when used to synthesize a third image. Therefore, these annotations can be used to evaluate the object detector.
[0038] Determining a systematic error can also be understood as determining only a single systematic error, such as the error concerning a single attribute or a single combination of attributes.
[0039] Preferably, before placing the object into the background image, the method further includes the following steps: • Rotate the object according to a predefined rotation and / or scale the object according to a predefined scale before placing the object into the background image.
[0040] Rotation and / or scaling can be considered another source of systematic error for object detectors. For example, object detectors may perform significantly worse for objects shown from behind or within a certain scale range.
[0041] Advantageously, this allows for the evaluation of even more sources of systematic error, and consequently provides more detailed reasoning about the behavior of the object detector.
[0042] Preferably, the Zero-1-to-3 method is used to rotate the object.
[0043] Advantageously, this allows for determining a 3D view of the object to be inserted into the background image without requiring external information about the camera that originally recorded the object. This allows for the collection of a large number of 2D images showing the object, while still being able to use these objects from different viewpoints within the method.
[0044] Preferably, the steps of the method are performed multiple times, wherein for each iteration of the step, the type of the object and / or at least one visual attribute and / or a predefined rotation and / or a predefined scale and / or the position in a third image are preferably randomly selected, and the detection performance is evaluated for each set of visual attributes and / or each set of rotations and / or each set of scales and / or each set of positions.
[0045] Advantageously, the evaluation can then be based on a wide variety of different scenarios and attributes, which in turn provides a more statistically meaningful assessment of the object detector. In other words, the results regarding systematic errors become even more reliable.
[0046] The impact of different object types, visual attributes, rotations, and scales on the detection performance of an object detector can be evaluated by determining the performance for different object types, visual attributes, rotations, and scales, where each object type, visual attribute, rotation, and scale can then be assigned a performance value corresponding to the respective visual attribute, rotation, or scale. However, different types of attributes, rotations, and scales can be represented based on different data types, i.e., based on discrete values (e.g., different colors) or based on continuous values (e.g., rotation angles). Performance values for discrete values are calculated directly, as performance values for each of the possible classes of a given attribute can be calculated. For continuous values, the possible values can be “discretized” into a predefined number of bins, similar to determining a histogram. For each bin, performance can then be evaluated. If the bin performance indicates systematic error (e.g., since the bin performance belongs to the average performance of the object detector), the range of bins can be presented as a source of systematic error (e.g., “the object detector systematically performs poorly on objects with z-angles ranging from -10° to 10°”).
[0047] The performance value corresponding to object type, visual attribute, rotation, or scale can be understood as the average performance when the remaining categories are integrated (i.e., marginalized). For example, the performance of the object detector can be determined for all generated images showing a car (object type), ignoring other parameters of generation (such as the car's color), namely visual attributes, rotation, and scale. This can be understood as keeping a single parameter (object type, visual attribute, rotation, scale) fixed while changing other parameters. Taking the average performance on such images results in only this parameter affecting the performance (in the example above, only the car is responsible for performance evaluation, while parameters such as color, rotation, and scale are "integrated").
[0048] If a statistical test determines that the performance is significantly worse than the performance determined for other attributes, then that performance can also be considered as a characterization of systematic error.
[0049] Preferably, if the object is magnified before being placed in an image of the background, the magnification is achieved by a machine learning system, preferably with a stable diffusion of 4x.
[0050] The inventors discovered that, advantageously, this magnification method provides a third image that more accurately reflects the true image, which in turn leads to better reliability of the determined assessment.
[0051] In another aspect, the present invention relates to a computer implementation method for training an object detector, comprising the following steps: • Determine the systematic error of the object detector according to any embodiment of the method for determining the systematic error of the object detector; • Train the object detector using images of objects that demonstrate object type and / or visual attributes and / or rotation and / or scale and / or position, where visual attributes and / or rotation and / or scale and / or position have been evaluated to characterize systematic errors.
[0052] Advantageously, training on images that exhibit visual attributes, rotation, and / or scale that are already considered sources of systematic error leads to a reduction in the systematic error of the object detector.
[0053] Images used for training can be manually annotated or automatically annotated using known automatic annotation methods.
[0054] If visual attributes, rotation, or scale are characterized by continuous values, then training images may include attributes, rotation, or scale taken from bins that are already considered sources of systematic error. Attached Figure Description
[0055] Embodiments of the invention will be discussed in more detail with reference to the following figures. The figures illustrate: Figure 1 It is a method for determining the systematic error of an object detector; Figure 2 It is a training system used to train object detectors; Figure 3 It is a control system that includes an object detector for controlling actuators in its environment; Figure 4 It is a control system that controls at least partially autonomous robots; Figure 5 It is a control system for controlling manufacturing machines. Detailed Implementation
[0056] Figure 1 An embodiment of a computer implementation method for determining the systematic error of an image-based object detector is shown.
[0057] In the first step (901), a first image of an object from a plurality of different types of objects is determined. This can be understood in particular as selecting a specific type of object and then selecting an image to display that object. For this purpose, the method may access a database of images, which can be evaluated by the type of object displayed in a particular image. Alternatively, an automated Internet search may preferably be performed on at least one image showing the desired object type, the image may be downloaded, and the downloaded image may be used as the first image.
[0058] In the second step (902), the edges or pose of the object are determined. If the object is an animal or a human, the pose is preferably determined. For other objects, the edges are preferably determined. However, the edges can also be determined for animals or humans. Therefore, whether to determine the edges or the pose for a given object type can be considered a hyperparameter of the method.
[0059] In the third step (903), at least one visual attribute is determined from a set of possible visual attributes of the object to be tested for systematic error.
[0060] In the fourth step (904), a second image is generated by using at least one visual attribute as a cue or for the generative machine learning system, thereby generating an image of an object displaying at least one visual attribute, wherein the generative machine learning system is additionally conditioned on the determined edges or poses of the object.
[0061] Generative machine learning systems can, in particular, be ControlNet configured to use edges or poses as conditional inputs. A first image can be provided as input to the generative machine learning system. Alternatively, objects can be "cut out" from the first image. This can be achieved by segmenting the object using a segmentation algorithm (e.g., segmenting everything) and determining the object's pixels. All other pixels can then be set to a uniform background, such as white.
[0062] Optionally, the method includes a fifth step (905) in which the object in the image generated by generative machine learning is rotated according to a predefined rotation before the object is placed in the background image, and / or the object is scaled according to a predefined scale.
[0063] In the sixth step (906), the object from the second image is placed into the image of the background scene to generate the third image.
[0064] You can also inject multiple objects into the background image by using the first four steps (901-904) or the first five steps (901-905) multiple times to identify different objects and place all objects into the background image.
[0065] In the seventh step (907), the object detector is evaluated on the third image to determine the performance of the object detector with respect to the third image.
[0066] Evaluating object detectors can specifically include calculating performance values, also known as “determining the performance of the object detector.” Performance values can be determined based on performance metrics that measure the “goodness” of the object detector’s performance. Common object detection metrics can be used, such as mean precision, mean precision, AP50, AP75, mean recall, mean recall, accuracy, or false negative rate.
[0067] The impact of different object types, visual attributes, rotation, and scale on the detection performance of an object detector can be evaluated by determining the performance for different object types, visual attributes, rotations, and scales, where each object type, visual attribute, rotation, and scale can then be assigned a performance value corresponding to the respective visual attribute, rotation, or scale. The performance value for discrete values can be directly calculated by computing the performance value for each of the possible classes of a given attribute. For continuous values, the possible values can be "discretized" into a predefined number of bins, similar to determining a histogram. For each bin, the performance can then be evaluated.
[0068] Preferably, these steps are repeated to determine the performance of multiple third images. Even more preferably, in each repetition of the steps, the type of object to be used, at least one visual attribute, the rotation and / or scale of the object are randomly selected.
[0069] Step seven (907) may also include a final evaluation of the sources of systematic error for a given object type, visual attribute, rotation, or scale. For example, the performance determined for a given parameter (i.e., given object type, visual attribute, rotation, or scale) can be compared to a predefined threshold, such as the average performance of the object detector across all test parameters. If the performance is worse than the predefined threshold, the corresponding parameter can be considered a source of systematic error.
[0070] Preferably, if the object is magnified before being placed in an image of the background, the magnification is achieved by a machine learning system, preferably with a stable diffusion of 4x.
[0071] Figure 2 An embodiment of a training system (140) is shown, which is used to train an object detector (60) using a training dataset (T). The training dataset (T) includes multiple input images used to train the object detector (60). The training dataset (T) also includes data for each input image. The expected output signal of ) ), which corresponds to the input image ( ) and characterize at least one object to be detected by the object detector (60), for example in the form of a bounding box and preferably the class of the object. Input image ( The object detector (60) includes multiple images depicting the object, which display parameters (object type, visual attributes, rotation, scale) that are considered to be sources of systematic error in the object detector (60). The object detector (60) can therefore be understood as existing in a trained form, in which the training system (140) fine-tunes the object detector (60) to mitigate systematic error.
[0072] The object detector (60) may be a neural network or include a neural network for locating and classifying objects in the provided image.
[0073] For training, the training data unit (150) accesses a computer-implemented database (St2) that provides a training dataset (T). The training data unit (150) preferably randomly selects at least one input image from the training dataset (T). ) and corresponding to the input image ( The expected output signal of ) ), and input image ( The image is transmitted to the object detector (60). The object detector (60) is based on the input image ( Determine the output signal ( ), where the output signal ( ) characterizes at least one object detection. For example, the output signal ( A detection object may include or consist of a bounding box, which represents the location of the detected object and the classification of the detected object.
[0074] Desired output signal ( ) and the determined output signal ( ) is transmitted to the modification unit (180).
[0075] Based on the desired output signal ( ) and the determined output signal ( ), modify unit (180) and then determine new parameters for object detector (60) ( For this purpose, the modified unit (180) uses a loss function to compare the desired output signal ( ) and the determined output signal ( The loss function determines the first loss value, which characterizes the determined output signal. The output signal deviates from the expected value. The degree of [missing information]. In the given embodiment, the negative log-likelihood function is used as the loss function. In alternative embodiments, other loss functions may also be conceived.
[0076] Furthermore, it can be envisioned that the determined output signal ( ) and the desired output signal ( Each of them includes multiple sub-signals, for example in tensor form, where the desired output signal ( The sub-signal corresponds to the determined output signal ( The first sub-signal represents the object with respect to the input image ( ). It can be envisioned that the first sub-signal represents the object with respect to the input image ( ). The probability of occurrence of a portion of the signal is given by the first sub-signal, while the second sub-signal characterizes the exact location of the object. If the determined output signal ( ) and the desired output signal ( If a signal comprises multiple corresponding sub-signals, then a second loss value is preferably determined for each corresponding sub-signal by means of a suitable loss function, and the determined second loss values are appropriately combined to form a first loss value, for example by means of a weighted sum.
[0077] The modification unit (180) determines new parameters based on the first loss value. In a given embodiment, this is accomplished using a gradient descent method, preferably stochastic gradient descent, Adam, or AdamW. In another embodiment, training may also be based on an evolutionary algorithm or a second-order method used to train the neural network.
[0078] In other preferred embodiments, the described training is iteratively repeated a predetermined number of times, or iteratively repeated until a first loss value falls below a predefined threshold. Alternatively or additionally, it is also conceivable that training terminates when the average first loss value on the test or validation dataset falls below a predefined threshold. In at least one iteration, new parameters (determined in the previous iteration) are used... ) is used as a parameter of the object detector (60) ).
[0079] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions which, when executed by the processor (145), cause the training system (140) to perform a training method according to one aspect of the invention.
[0080] Figure 3 An embodiment of a control system (40) is shown, which includes an object detector (60) for controlling the actuator (10) in its environment (20). The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably uniformly spaced time points, an optical sensor (30) senses the condition of the actuator system. The sensor (30) may include a plurality of optical sensors. The output signal (S) of the sensor (30) (or, in the case where the sensor (30) includes a plurality of sensors, the output signal (S) of each sensor) is transmitted to the control system (40), the output signal (S) encoding the sensed condition.
[0081] Thus, the control system (40) receives the sensor signal (S) stream. It then calculates a series of control signals (A) based on the sensor signal (S) stream, which are then transmitted to the actuator (10).
[0082] The control system (40) receives the sensor signal (S) stream from the sensor (30) in an optional receiving unit (50). The receiving unit (50) converts the sensor signal (S) into an input image ( Alternatively, in the absence of a receiving unit (50), each sensor signal (S) can be directly used as the input image. Input image () For example, it can be given as an extract of the sensor signal (S). Alternatively, the sensor signal (S) can be processed to generate an input image ( In other words, the input image is provided based on the sensor signal (S). ).
[0083] Input image ( Then it is passed to the object detector (60).
[0084] The object detector (60) is composed of parameters ( ) parameterization, and these parameters are stored in a parameter storage device ( ) and by parameter storage device ( )supply.
[0085] The object detector (60) receives the input signal ( Determine the output signal in ) Output signal () This includes assigning object detection to the input image ( ) or characterize the input image ( The output signal does not contain information about the object. The signal is transmitted to an optional conversion unit (80), which outputs the signal. The signal is converted into a control signal (A). The control signal (A) is then transmitted to the actuator (10) for corresponding control of the actuator (10). Alternatively, the output signal (A) is converted into a control signal (A). ) can be directly used as a control signal (A).
[0086] The actuator (10) receives a control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include control logic that transforms the control signal (A) into another control signal, which is then used to control the actuator (10).
[0087] In another embodiment, the control system (40) may include a sensor (30). In yet another embodiment, the control system (40) may alternatively or additionally include an actuator (10).
[0088] In yet another embodiment, it is conceivable that the control system (40) controls the display (10a) in addition to the actuator (10) or the actuator (10).
[0089] Furthermore, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) thereon storing instructions which, if executed, cause the control system (40) to perform the method according to aspects of the invention.
[0090] Figure 4 An embodiment is shown in which the control system (40) is used to control at least a partially autonomous robot (e.g., at least a partially autonomous vehicle (100)).
[0091] The sensor (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more lidar sensors. Some or all of these sensors are preferably, but not necessarily, integrated into the vehicle (100).
[0092] The object detector (60) can be configured to be based on the input image ( Detects objects at least partially near the autonomous robot. Output signal ( This can include information characterizing the location of the object near at least part of the autonomous robot. Control signals (A) can then be determined based on this information, for example, to avoid collisions with the detected object.
[0093] The actuator (10), preferably integrated into the vehicle (100), can be provided by the vehicle's (100) brakes, propulsion system, engine, transmission system, or steering mechanism. A control signal (A) can be determined such that the actuator (10) is controlled to prevent the vehicle (100) from colliding with a detected object. The detected objects can also be classified according to what the object detector (60) considers them to be most likely, such as a pedestrian or a tree, and the control signal (A) can be determined based on the classification.
[0094] Alternatively or additionally, the control signal (A) may also be used to control the display (10a), which, for example, is used to display objects detected by the object detector (60). It is also conceivable that the control signal (A) could control the display (10a) such that if the vehicle (100) approaches and collides with at least one of the detected objects, the display generates a warning signal. The warning signal could be an audible warning and / or tactile signal, such as vibration of the vehicle's steering wheel.
[0095] In another embodiment, at least a partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving, or walking. The mobile robot may, in particular, be at least a partially autonomous lawnmower or at least a partially autonomous cleaning robot. In all the above embodiments, a control signal (A) may be determined such that the propulsion unit and / or steering device and / or brakes of the mobile robot are controlled to enable the mobile robot to avoid collisions with the identified object.
[0096] In another embodiment, at least a portion of the autonomous robot may be provided by a gardening robot (not shown) that uses sensors (30) (preferably optical sensors) to determine the state of the plants in the environment (20). Actuators (10) may control nozzles and / or cutting devices (e.g., blades) for spraying liquid. Based on the identified species and / or identified state of the plant, control signals (A) may be determined to cause the actuators (10) to spray and / or cut the plant with an appropriate amount of liquid.
[0097] Figure 5 An embodiment is shown in which a control system (40) controls a manufacturing machine (11) (e.g., a press, cutter, gun drill, or jig) of a manufacturing system (200) (e.g., as part of a production line). The manufacturing machine may include transport equipment, such as a conveyor belt or assembly line, that moves the manufactured product (12). The control system (40) controls an actuator (10), which in turn controls the manufacturing machine (11).
[0098] The sensor (30) can be provided by an optical sensor that captures, for example, the properties of the manufactured product (12).
[0099] The object detector (60) can determine the position of the manufactured product (12) relative to the transport equipment. The actuator (10) can then be controlled based on the determined position of the manufactured product (12) for subsequent manufacturing steps of the manufactured product (12). For example, the actuator (10) can be controlled to cut the manufactured product at a specific location on the manufactured product itself. Alternatively, it is conceivable that the object detector (60) classifies whether the manufactured product is broken and / or exhibits defects. The actuator (10) can then be controlled to remove the manufactured product from the transport equipment.
[0100] The term "computer" can be understood as any device used to process predefined computational rules. These computational rules can be in the form of software, hardware, or a combination of both.
[0101] Typically, "multiple" can be understood as being indexed, meaning each element in the multiple is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the multiple. Preferably, if the multiple includes... There are elements, among which If the number of elements in a plurality is a given number, then the elements are assigned from... arrive Integers. It can also be understood that multiple elements can be accessed through their indices.
[0102] A list and / or combination of different elements is understood in particular to expose each element individually, a subset of any combination of the elements of the list, or all the elements.
Claims
1. A computer-based method for determining the systematic error of an image-based object detector, comprising the following steps: • Determine (901) the first image of an object from multiple different types of objects; • Determine (902) the edge of the object or the pose of the object; • Determine (903) at least one visual attribute from a set of possible visual attributes of the object to be tested for systematic error; • A second image is generated by a generative machine learning system using the at least one visual attribute as a cue or for the generative machine learning system, thereby generating an image of the object displaying the at least one visual attribute, wherein the generative machine learning system is additionally conditioned on the determined edges or poses of the object. • The object from the second image is placed (906) into an image of the background scene to generate a third image; • Evaluate the object detector on the third image (907) and use the evaluation to determine the systematic error of the object detector.
2. The method according to claim 1, wherein, Before placing the object into the background image, the method further includes the following steps: • Before placing the object into the background image, rotate (905) the object according to a predefined rotation and / or scale the object according to a predefined scale.
3. The method according to claim 2, wherein, Use the Zero-1-to-3 method to rotate the object.
4. The method according to any one of claims 1 to 3, wherein, The steps are performed multiple times, wherein, for each iteration of the steps, preferably at least one visual attribute and / or a predefined rotation and / or a predefined scale and / or the position of the object in the third image and / or the type of the object are randomly selected, and the detection performance is evaluated for each visual attribute and / or for each set of rotations and / or for each set of scales and / or for each set of positions and / or for each object type.
5. The method according to claim 4, wherein, When the detection performance corresponding to a visual attribute and / or a set of rotations and / or a set of scales and / or a set of positions and / or object types is equal to or lower than a predefined threshold, the visual attribute and / or the set of rotations and / or the set of scales and / or the set of positions are provided as a source of systematic error.
6. The method according to claim 5, wherein, The threshold is the average detection performance of the object detector.
7. The method according to any one of claims 1 to 6, wherein, The generative machine learning system is ControlNet.
8. The method according to any one of claims 1 to 7, wherein, The object is placed into the background image using stable diffusion repair.
9. The method according to any one of claims 1 to 8, wherein, Determining the first image includes segmenting the object from another image and providing the object in front of a uniform background, preferably a white background, as the first image.
10. The method according to any one of claims 1 to 9, wherein, The object is magnified before being placed in an image of the background, wherein the magnification is achieved by a machine learning system, preferably with a stable diffusion of 4x.
11. A computer implementation method for training an object detector, comprising the following steps: • Determine the systematic error of the object detector according to any one of claims 1 to 10; • The object detector is trained using images including objects that demonstrate visual attributes and / or rotation and / or scale and / or position and / or object type, wherein the visual attributes and / or rotation and / or scale and / or position and / or object type have been evaluated to characterize systematic errors.
12. A training system (140) configured to implement the training method according to claim 11.
13. A control system (40) configured to use an object detector (60) to determine a control signal (A), the object detector being trained using the method according to claim 11, wherein the control signal (A) is configured to control an actuator (10) and / or a display (10a).
14. A computer program configured to cause a computer to perform the method and all steps thereof according to any one of claims 1 to 11 when the computer program is implemented by a processor (45, 145).
15. A machine-readable storage medium (46, 146) having a computer program as claimed in claim 14 stored thereon.