System and method for the automated determination of material properties of a component

Abstract

The invention relates to a system and a method for the automated determination of material properties of a component, particularly for a vehicle. The system comprises an image acquisition module with at least one image acquisition device configured to capture image data from one or more images of the component; a sensor module with at least one biomimetic sensor that captures tactile sensor data of the component;a data processing module comprising an image processing module, an AI model, and a language model, wherein the image processing module is trained to perform a segmentation of the component on the image representation based on the image data, wherein the AI ​​model comprises an energy-based model and is trained to analyze and determine the material properties of the component based on the image data and the sensor data, wherein the language model is trained to interpret input prompts in the form of text or voice messages and forward them to the AI ​​model; and a user interface trained to capture the input prompts and to display the analysis result of the material determination as a text message or visual representation.

Description

[0001] The invention relates to a system, method and computer program product for the automated determination of material properties of a component, in particular for a vehicle.

[0002] The mechanical components of a vehicle can consist of a variety of materials, each with specific properties that determine its suitability for certain applications.

[0003] A typical material is steel, which is characterized by high strength, toughness, good formability and weldability, and is also relatively inexpensive. Steel is frequently used for chassis components, body components, and axles. Steel is easily recognizable by its characteristic surface and its behavior under mechanical stress, e.g., its typical deformation pattern.

[0004] Aluminum is lightweight, corrosion-resistant, has good thermal conductivity, and relatively high strength-to-weight ratio. It is frequently used for engine blocks, wheel suspensions, and body components. The matte, silvery surface of aluminum and the type of oxidation provide clues about the material.

[0005] Plastics (polymers) are lightweight, malleable, corrosion-resistant, and inexpensive. They can be manufactured in a wide range of strengths and flexibilitys and are typically used for trim panels, interior components, cable sheathing, and some structural parts. Plastic surfaces are relatively easy to identify due to their texture and color.

[0006] Carbon fiber reinforced polymers (CFRP) are characterized by extremely high strength and stiffness at low weight, but are expensive and more difficult to process. CFRP materials are used in high-performance vehicles, particularly in motorsports, where CFRP components are used for body parts, drive shafts, and other highly stressed components. CFRP materials have a characteristic woven structure that is easily recognizable visually.

[0007] Titanium has high strength and corrosion resistance as well as low weight, but it is very expensive. Titanium is used where low weight and high strength are required, e.g., for special screws, fasteners, and exhaust systems. Titanium often has a slightly matte, metallic surface and can be identified by its color and tarnish.

[0008] Cast iron is characterized by high compressive strength, good damping properties, and good machinability, but it is brittle. Cast iron is used for engine blocks, cylinder heads, and brake components. Its typical rough surface and fracture behavior can indicate that it is cast iron.

[0009] The ability to identify the materials of a component using images allows engineers, for example, to quickly assess the material properties and thus the potential load-bearing capacity, durability, and processability of a component. Engineers who are not on-site can use the images for material analysis, which is particularly advantageous for remote maintenance, inspections, or remote engineering. Early identification of the materials used in a component can prevent unnecessary material testing, saving time and money. Furthermore, image analysis can be used in production for quality assurance to ensure that the correct materials have been used. In the event of a component failure, image analysis can help identify the cause of the damage, for example, by detecting material fatigue or corrosion.

[0010] The identification of materials through imaging techniques can therefore be a powerful tool for engineers to optimize processes, support maintenance work, and improve the quality and safety of vehicles.

[0011] However, the error rate of human observers when visually identifying materials from images is relatively high. For example, confusing materials such as steel and aluminum is a common problem, and there are several reasons for this confusion.

[0012] Steel and aluminum can exhibit similar silvery to gray hues and metallic luster, especially when polished or machined. Without specialized knowledge or additional information, it is difficult for the human observer to distinguish them based on appearance alone. Furthermore, both materials can have similar surface textures or coatings, further complicating the differentiation.

[0013] A key difference between steel and aluminum is density, as steel is about three times heavier than aluminum. However, this property cannot be grasped simply by looking at a picture. An observer who isn't physically holding the component cannot use this important distinguishing feature. Furthermore, unlike aluminum, steel is magnetic, which also cannot be seen in a picture.

[0014] Furthermore, there are differences in roughness and sliding friction between steel and aluminum. Steel surfaces tend to be rougher, especially when unpolished, as steel is often used for applications requiring a harder surface. Aluminum surfaces are generally smoother, particularly when anodized or polished, because aluminum is easier to machine but also more susceptible to scratches.

[0015] In terms of sliding friction, steel generally has higher sliding friction than aluminum, especially on rough surfaces. This means it is more difficult to move objects on a steel surface. Aluminum, on the other hand, has lower sliding friction due to its smoother surface structure, which makes it advantageous for certain applications, such as machine parts.

[0016] Many observers, especially those without specialized training in materials science, find it difficult to reliably identify materials based solely on visual indicators. Human observers tend to draw conclusions based on prior experience or expectations. For example, someone who works with aluminum more often than with steel might mistakenly assume it's aluminum due to their expectations.

[0017] Especially with poor image quality, subtle differences in surface texture and gloss become less apparent, making material identification more difficult. Furthermore, certain viewing angles or lighting conditions can alter the appearance of materials and lead to misinterpretations.

[0018] Misidentification of materials due to confusion can lead to incorrect assumptions about a component's strength, weight, corrosion resistance, and other important properties. Inaccurate material assumptions can result in faulty calculations or unsuitable operating conditions. In safety-critical applications, incorrect material identification can have potentially harmful consequences.

[0019] WO 2019 / 158809 A1 discloses a system and a method for improving user immersion in a mixed-reality mode of a head-mounted display device. The system includes a camera that captures a sequence of images of the real environment. The image sequence is analyzed to determine the spatial geometry of real objects in the real environment and the material categories of the real objects. A sequence of mixed-reality images is generated based on the spatial geometry and material category of at least one real object and includes at least one virtual object whose visual behavior emulates at least one material property associated with the material category of the at least one real object.

[0020] German patent DE 10 2019 211 526 A1 relates to a method and a system for generating an enhanced image of a target object. Image data and tactile data of the target object are acquired. The tactile data characterizes the mechanical resistance of at least a portion of the target object to a force applied to it. The enhanced image is then generated by fusing the image data and the tactile data and / or by deriving a classification of the respective portion from this.

[0021] DE 10 2023 127 295 A1 relates to a method for examining a component of a turbomachine, in particular an aircraft engine, comprising the steps of: moving the sensor to detect the component along a sensor path relative to the component; detecting the areas of the component lying on the sensor path, wherein the sensor is robot-guided with control signals along the sensor path and the control signals are generated using AI.

[0022] The object of the invention is the development of a system and a method for the automated and precise detection and classification of materials of components, in particular of a vehicle, based on image recordings. The system should be able to correctly identify various materials such as steel, aluminum, titanium and plastics based on their optical properties and other relevant characteristics.

[0023] This problem is solved according to the invention with respect to a system by the features of claim 1, with respect to a method by the features of claim 8, and with respect to a computer program product by the features of claim 13. The further claims relate to preferred embodiments of the invention.

[0024] The system according to the invention offers flexible and precise material identification through the integration of image acquisition and tactile measurement of a component. It allows the use of various image acquisition devices, such as conventional cameras or VR glasses, which can be selected depending on the specific task. The integration of tactile measurements results in significantly improved material recognition, far exceeding the capabilities of purely visual methods. The energy-based model is characterized by high accuracy in material identification, as it has been trained on a large number of components under varying conditions and with different component geometries. The combination of these technologies enables a reliable and accurate solution for material identification in diverse application areas.

[0025] The invention can be used, for example, for quality control in production to ensure that the correct materials have been used. It is also useful in the maintenance and inspection of components, particularly in hard-to-reach areas, for carrying out precise material analyses. In the recycling sector, the invention helps to sort waste materials by enabling accurate material identification.

[0026] According to a first aspect, the invention provides a method for the automated determination of material properties of a component, particularly for a vehicle. The system comprises an image acquisition module with at least one image acquisition device configured to acquire image data from one or more image representations of the component under various conditions, wherein the component may be arranged in a larger assembly; a sensor module with at least one biomimetic sensor that acquires tactile sensor data of the component, including roughness and friction properties of the at least one material of the component;A data processing module comprising an image processing module, an AI model, and a language model, wherein the image processing module is trained to perform a segmentation of the component on the image representation based on the captured image data, wherein the AI ​​model comprises an energy-based model and is trained to analyze and determine the material properties of the component based on the image data and the sensor data, wherein the language model is trained to interpret input prompts in the form of text or voice messages and forward them to the AI ​​model; and a user interface trained to capture the input prompts and forward them to the data processing module and to display the analysis result of the material determination as a text message or visual representation.

[0027] In a further training course, it is planned that the image recording device is designed as a high-resolution camera, smartphone or VR glasses in order to capture 2D or 3D representations or immersive 3D representations of the component in the form of image data.

[0028] In an advantageous embodiment, the AI ​​model is implemented as a Joint Embedding Predictive Architecture (JEPA), Latent Variable Energy-Based Model (LVEBM) or Latent Variable Generative Energy-Based Model (LVGEBM).

[0029] In a further embodiment, it is provided that the biomimetic sensor is designed to detect stick-slip behavior (stick-slip sensing), and that the acquired sensor data provide information about the friction properties of the material and are used to determine latent variables that are crucial for the correct classification of the material by the AI ​​model.

[0030] In particular, the AI ​​model was trained with a second training dataset, which includes annotated images of components stored in a database and measurement results from tactile measurements of different materials, where the annotations contain in particular information about specific material properties and / or different lighting situations, and where during the training of the AI ​​model the latent variables based on the tactile measurements were learned by the AI ​​model to correctly assign the right material.

[0031] Advantageously, the image processing module uses a segmentation algorithm, specifically implemented by the "Segment Anything Model" (SAM), where the segmentation algorithm was trained with an initial training dataset comprising a variety of images of components, particularly of vehicles, especially images of powertrain components such as gearbox housings, clutches, axles and exhaust pipes, brake components such as brake discs, brake pads and brake lines, cooling components such as coolant reservoirs, chassis components such as shock absorbers, engine components such as cylinder heads, pistons, crankshafts and camshafts, and body components such as bumpers, doors and fenders.

[0032] In particular, the user interface includes a touchscreen, a desktop computer or a mobile device to enable interactive user interaction.

[0033] According to a second aspect, the invention provides a method for the automated determination of material properties of a component, particularly for a vehicle. The method comprises the following steps: - Capturing image data from one or more image representations of the component using an image capture device; - Acquiring tactile sensor data of the component with a biomimetic sensor, wherein the tactile sensor data is acquired by a biomimetic sensor designed to detect stick-slip behavior (stick-slip sensing), wherein the acquired sensor data provides information about the friction properties of the material and is used to determine latent variables that are crucial for the correct assignment of the material by an AI model; - Entering a prompt for material identification into a user interface; - Processing the input prompt using a language model; - Segmenting the component from the image data of the captured image representation of the component using an image processing module; - Analyzing and determining the material of the component using an AI model that processes image data and tactile sensor data in combination, whereby the material determination is carried out by an energy-based model of the AI ​​model that is implemented as a Joint Embedding Predictive Architecture (JEPA) or Latent Variable Energy-Based Model (LVEBM); - Displaying the analysis result of the material determination on the user interface in the form of a text message or a visual representation.

[0034] In a further training course, it is planned that the image capture will be carried out using a high-resolution camera, a smartphone or VR glasses in order to capture 2D or 3D representations or immersive 3D representations of the component in the form of image data.

[0035] In particular, the AI ​​model is trained with a second training dataset, which includes annotated images of components stored in a database and measurement results from tactile measurements of different materials, wherein the annotations contain in particular information about specific material properties and / or different lighting situations, and wherein during the training of the AI ​​model the latent variables based on the tactile measurements are learned by the AI ​​model for the later correct assignment of the correct material.

[0036] Advantageously, the image processing module uses a segmentation algorithm, in particular implemented by the "Segment Anything Model" (SAM), and the segmentation algorithm is trained with an initial training dataset comprising a variety of images of components, especially vehicles, in particular images of powertrain components such as gearbox housings, clutches, axles and exhaust pipes, brake components such as brake discs, brake pads and brake lines, cooling components such as coolant reservoirs, chassis components such as shock absorbers, engine components such as cylinder heads, pistons, crankshafts and camshafts, and body components such as bumpers, doors and fenders.

[0037] In particular, the input prompt for material determination of the component is in the form of a text or voice message and is processed by the language model, whereby the display of the analysis result of the material determination contains additional contextual information taken from the annotations of the images used in the first training dataset or second training dataset.

[0038] According to a third aspect, the invention provides a computer program product with an executable program code that is configured to perform the method according to the second aspect when executed.

[0039] The invention will now be explained in more detail with reference to an embodiment shown in the drawing.

[0040] It shows: Fig. 1 a block diagram to illustrate an embodiment of a system according to the invention; Fig. 2 a flowchart to explain the individual process steps of a process according to the invention; Fig. 3 a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

[0041] Additional features, aspects and advantages of the invention or its embodiments become apparent from the detailed description in conjunction with the claims.

[0042] Fig. Figure 1 shows a system 100 according to the invention for the automated prediction of material properties of a component 20, in particular for a vehicle. The system 100 comprises an image acquisition module 200, a sensor module 300, a data processing module 400 with an AI model 450, a user interface 500, and a database 700. In particular, the image acquisition module 200, the sensor module 300, the data processing module 300, and the database 700 can each be equipped with a storage unit and / or a processor.

[0043] In the context of the invention, a "module" is defined as a self-contained, specialized unit of software and / or hardware components. A module is designed to perform a specific function or task and is independent and self-contained; that is, it accepts specific inputs, performs internal processing, and then delivers specific outputs or results. A module can communicate with other modules or components via interfaces. These interfaces determine how data or commands are input into the module and how results or information are output.

[0044] In the context of the invention, a "processor" can be, for example, a machine or an electronic circuit. A processor can, in particular, be a central processing unit (CPU), a microprocessor, or a microcontroller, such as an application-specific integrated circuit or a digital signal processor, optionally in combination with a memory unit for storing program instructions. A processor can also be a virtualized processor, a virtual machine, or a soft CPU. It can, for example, also be a programmable processor equipped with configuration steps for executing the method according to the invention, or configured with configuration steps such that the programmable processor implements the features of the method, the modules, or other aspects and / or partial aspects of the invention.In particular, the processor can contain highly parallel computing units and powerful graphics modules.

[0045] In the context of the invention, a "storage unit" or "storage module" and the like can refer, for example, to volatile memory in the form of random access memory (RAM), permanent storage such as a hard drive or data carrier, or, for example, a replaceable storage module. The storage module can also be a cloud storage solution.

[0046] In particular, the Data Processing Module 400 and the Database 700 can be integrated into a cloud computing infrastructure. A cloud computing infrastructure offers the ability to scale resources up or down as needed, allowing computing power, storage space, or network resources to be easily adapted to changing requirements. This scalability enables cost optimization and efficient resource allocation without large investments in hardware. Furthermore, users can access applications and data from anywhere with internet access. Cloud computing also offers high flexibility in software deployment, enabling applications to be deployed and updated quickly and without interruption. Cryptographic encryption methods can also be used to protect the connection to the cloud computing infrastructure via a mobile network.

[0047] Furthermore, communication links are provided for the exchange and transmission of data between the individual modules, in particular as wireless communication links, e.g. as mobile communication links (e.g. 4G LTE, 5G, 6G) and / or as near-field communication links, e.g. Bluetooth. ® , Ethernet, NFC (Near Field Communication) or Wi-Fi ® are trained.

[0048] There are a number of vehicle components (20) for which material analysis may be of particular interest. These include, in particular: a) Exhaust manifold - Materials: Stainless steel, cast iron, titanium - Reason for material selection: The exhaust manifold must withstand high temperatures and corrosive exhaust gases. Stainless steel offers corrosion and heat resistance, while titanium additionally offers lower weight with high strength. b) Engine block - Materials: Gray cast iron, aluminum, composite materials - Reason for material selection: The choice of material affects the thermal conductivity, strength, and weight of the motor. Aluminum is lighter, but cast iron has better damping properties and higher strength. c) Piston - Materials: Aluminum alloys, steel - Reason for material selection: Pistons must withstand extreme temperatures and pressures. Aluminum alloys offer a good combination of weight and thermal conductivity, while steel can be used in high-performance engines for added strength. d) Body components (e.g. doors, hood) - Materials: Steel, aluminum, carbon fiber - Reason for material selection: The body must be stable, safe, and as light as possible. Aluminum and carbon fiber allow for weight reduction, while steel is often used because of its high strength and availability. e) Brake discs - Materials: Gray cast iron, carbon fiber composites, ceramics - Reason for material selection: Brake discs must withstand high friction and heat loads. Carbon fiber composites and ceramics offer low weight and high heat resistance, while gray cast iron is widely used due to its high thermal conductivity and affordability. f) Chassis components (e.g. wishbones, springs) - Materials: Steel, aluminum, composite materials - Reason for material selection: Chassis components must be robust and durable to withstand the stresses encountered during driving. Aluminum and composite materials can reduce weight, while steel is traditionally used for its strength. g) Cardan shaft - Materials: Steel, aluminum, carbon fiber - Reason for the material selection: The driveshaft transmits the torque from the gearbox to the axles. Aluminum and carbon fiber allow for weight reduction, which can improve efficiency, while steel is traditionally used for its strength. h) Shock absorber - Materials: Steel, aluminum, composite materials - Reason for material selection: Shock absorbers must be sufficiently resistant and stable to absorb shocks and vibrations. Aluminum and composite materials can reduce weight and improve responsiveness. i) Rims - Materials: Aluminum alloys, steel, magnesium, carbon fiber - Reason for material selection: Rims must be lightweight and strong to optimize handling and performance. Aluminum alloys are lightweight and cost-effective, while carbon fiber is extremely lightweight and strong, but more expensive. j) Radiator and heat exchanger - Materials: Aluminum, copper, plastic composites. - Reason for the material selection: Heat sinks must be able to dissipate heat efficiently. Aluminum and copper are excellent heat conductors, with aluminum being preferred due to its low weight.

[0049] In a vehicle with an electric motor, there are other components where the choice of materials is of great importance. These components differ significantly from those in combustion engines, as they must meet different physical requirements. Some examples are listed below: a) Rotor and stator - Materials: Electrical steel, silicon steel, copper - Reason for material selection: The rotor and stator are the central components of an electric motor. The materials must exhibit high magnetic permeability and low losses to maximize the motor's efficiency. Silicon steel has lower hysteresis losses than conventional steel, which increases the energy efficiency of electric motors and transformers. Copper is used for the windings because of its excellent conductivity. b) Windings (conductors in stator and rotor) - Materials: Copper, Aluminum - Reason for the material selection: The windings in an electric motor must have very good electrical conductivity to minimize electrical resistance and maximize efficiency. Copper is frequently used, although aluminum is sometimes employed as a more cost-effective and lighter alternative. c) Housing of the electric motor - Materials: Aluminum, plastic, composite materials - Reason for material selection: The housing of an electric motor must be both lightweight and stable, while simultaneously allowing for good heat dissipation. Aluminum is widely used due to its low weight and good thermal conductivity. In some cases, plastics or composite materials are also used to further reduce weight. d) Cooling systems - Materials: Aluminum, copper, plastics. - Reason for material selection: Electric motors, especially in high-performance applications, require an effective cooling system to dissipate the heat generated. Aluminum and copper are preferred materials for heat sinks and heat exchangers due to their high thermal conductivity. Plastic can be used for cooling channels where the mechanical stress is lower.

[0050] The choice of materials, especially for these components, is crucial for the performance, efficiency and longevity of the electric motor and thus of the entire electric vehicle.

[0051] Within the scope of the invention, component 20 can be part of a larger assembly or a more comprehensive system. For example, component 20 could represent a component such as a rotor, a shaft, a housing, or another functional unit. A larger assembly is a more complex unit comprising several components, including component 20. This assembly can, in turn, be part of an even more comprehensive system, such as a motor, a transmission, an electronic control unit, or another complex unit.

[0052] The image acquisition module 200 is responsible for acquiring an image of the component 20 to be analyzed. It can be implemented in various ways according to specific requirements. The image acquisition module 200 comprises at least one image acquisition device 220 for acquiring an image of the component 20 under investigation. Typically, an image of the component 20 is acquired in its installed state within such a more complex assembly. For example, a rotor (component 20) may be installed in an electric motor (larger assembly), where its material properties are crucial for the overall system. Furthermore, the invention may provide that the material analysis is not limited to a single component 20 in a single image, but encompasses a multitude of other components that interact within the entire assembly.

[0053] The image acquisition device 220 can be configured as a high-resolution camera, a smartphone, or VR (Virtual Reality, VR) glasses for immersive 3D representations. A user, for example an engineer, can use a suitable image acquisition device 220 to capture image data 250 of an image representation of the component 20 or an assembly containing the component 20, either as a conventional 2D representation or as an enhanced 3D representation, for example with a 3D camera or VR glasses.

[0054] A high-resolution camera is characterized by very high image quality and detail accuracy, as it captures image data (250) with a high pixel count. This enables the capture of the finest details of component 20, which is particularly important for analyzing surface structure and potential material properties. These cameras are ideal when precise visual information is needed to detect even the smallest differences in material properties.

[0055] In addition to the usual two-dimensional information, a 3D camera also captures the spatial depth of component 20. This is achieved by using multiple lenses or special sensors that measure the distance of various points on component 20 from the camera. This technology enables a three-dimensional representation of component 20, allowing for a more precise analysis of the shape, structure, and spatial relationships of its surfaces. This is particularly useful when the geometry of component 20 is relevant for material determination.

[0056] A VR headset is a device that allows the user to immerse themselves in a computer-generated three-dimensional environment. The VR headset consists of a display directly in front of the user's eyes and is equipped with sensors that track head movements in real time. This adjusts the user's field of vision so they can look freely around the virtual environment as if they were actually there.

[0057] VR headsets use high-resolution displays, often with OLED or LCD technology, which deliver a slightly offset image to each eye separately. This creates the impression of depth and spatial perception, similar to human vision in the real world.

[0058] VR headsets are equipped with gyroscopes, accelerometers, and sometimes external cameras or lidar sensors that track the position and movement of the user's head. This data is used to adjust the virtual environment so that the user's view of the virtual world remains synchronized with their movements.

[0059] To optimally display component 20 and avoid distortions, VR glasses are equipped with special lenses that magnify the image of component 20 and project it correctly onto the user's eye. These lenses play a crucial role in a realistic and comfortable viewing experience.

[0060] VR headsets often include handheld controllers or even hand-tracking technologies that allow the user to interact with the virtual environment. This can include grabbing, dragging, or moving objects within the VR world.

[0061] The invention utilizes VR glasses to provide the user with an immersive viewing experience of component 20 within a virtual environment. This allows the component 20 to be viewed from various angles and under different lighting conditions. The 3D representation and interactivity of the VR environment can significantly facilitate the recognition of surface structures and geometries, and thus the analysis of material properties. By integrating additional data such as markings and annotations, targeted information can be displayed directly in the user's field of vision. This augmented reality, in particular, enables the simulation of different lighting conditions, thereby improving material identification.Especially with complex assemblies, VR glasses can be a valuable aid in better understanding and visualizing the interaction of different materials within a system.

[0062] Since the representation of depth and precise surface structure of a component is limited in a conventional 2D image, VR headsets can be very useful for certain applications, as they create an interactive environment in which the user can simulate different scenarios and surface conditions. However, for less demanding applications, a normal static image is often sufficient.

[0063] Furthermore, the type of lighting plays a crucial role in the image acquisition of component 20. Depending on the specific requirements, the lighting can be adjusted to highlight or hide certain reflections.

[0064] On metallic or glossy surfaces, targeted illumination can create reflections that aid in the analysis of the material properties. For example, selectively directed light sources can produce highlights or reflections that reveal characteristic properties of the component material.

[0065] In other cases, it may be necessary to minimize reflections to ensure a clear view of the surface structure of component 20. This can be achieved by using diffuse light or polarizing filters, which reduce interfering reflections and ensure uniform illumination of component 20. Such illumination makes it possible to better detect details such as scratches, cracks, or irregularities.

[0066] Additionally, the lighting conditions in the image of component 20 can be simulated retrospectively to create different lighting situations. This includes adjusting the image contrast, brightness, and other lighting parameters. This post-processing of the image makes it possible to consider the effects of different lighting conditions on image quality and the recognition of material properties. As a result, the AI ​​model 450 can react better to changing lighting conditions and improve material recognition under various lighting conditions.

[0067] Furthermore, text and / or speech data can be used to improve image processing and image simulation. This additional data provides supplementary information about component 20, the component surface, or the current lighting conditions and contributes to increasing the accuracy of the image simulation processes.

[0068] The text data can contain detailed descriptions of component 20 and information on surface properties or lighting conditions. This data serves as supplementary information that provides context for the image data 250. For example, it can include details on material properties or specific surface irregularities that must be considered during image processing. The text data helps the AI ​​model 450 to perform a more accurate adjustment and simulation of image quality by providing additional input factors for image correction and analysis.

[0069] In addition to textual data, linguistic descriptions of the component, its surface properties, or the lighting conditions can also be used. These linguistic descriptions can be converted into textual data and interpreted using language models. They also contain additional information about material properties or specific details that are important for accurate image simulation and analysis.

[0070] The integration of text and speech descriptions can supplement the visual information to enable a more comprehensive and accurate simulation of the pictorial representation of component 20. This improves the ability of the AI ​​model 450 to identify and analyze material properties, particularly in cases where the visual information alone is incomplete.

[0071] Furthermore, different materials exhibit differences in roughness and sliding friction, which can be detected by tactile measurements. For example, steel and aluminum can be distinguished by measuring roughness and sliding friction when visual characteristics alone are insufficient. The system 100 described in the invention utilizes these tactile properties to enable more precise material identification.

[0072] For this purpose, the sensor module 300 comprises at least one biomimetic sensor 320 to detect the tactile properties of the component 20's material. The biomimetic sensor 320 simulates the touch of a human finger in the form of a stick-slip motion by sliding across the material surface of the component 20 to measure the sliding friction force and the roughness of the material. The sensor data 350 from this measurement provide additional information about the surface properties, which is incorporated into the material determination. According to the invention, this sliding friction measurement is used as a latent variable in the energy-based AI model 450.

[0073] Stick-slip sensing is a specialized technique frequently used in biomimetic sensors. This method measures the differences between the stick and slip phases during contact with a surface. As the surface moves or slips, the sensor surface generates characteristic signals that can be analyzed to determine material properties such as roughness, hardness, and surface texture.

[0074] The sensor data 350 recorded by the biomimetic sensor 320 are forwarded to the data processing module 400, which evaluates the sensor data 350 to draw conclusions about the material properties of the component 20. This evaluation can be carried out in different ways, for example by comparing the measured frictional forces with known reference values ​​or by applying special algorithms for pattern recognition.

[0075] Within the scope of the invention, the sliding friction values ​​measured with the biomimetic sensor 320 contribute to a more accurate material determination by providing additional information about the surface properties and the behavior of the material under friction. By combining these tactile measurement results with image-based information, the material determination can be carried out more precisely and comprehensively.

[0076] The data processing module 400 includes an image processing module 430, which contains algorithms for segmenting the acquired image data 250. The image processing module 430 segments the acquired image data 250 of the image representation of component 20 in order to separate the relevant component 20 from its background in the image. For this purpose, a sophisticated segmentation algorithm such as SAM (Segment Anything Model) is used.

[0077] Segment Anything Model (SAM) is a powerful image segmentation algorithm designed to accurately detect and isolate objects in images regardless of their size, shape, or position. SAM was developed by Meta AI. ® SAM is designed and trained to segment a wide variety of objects in diverse image contexts. It uses a combination of convolutional neural networks (CNNs) and transformer models to analyze image data and precisely define the boundaries of relevant objects. It is capable of separating complex backgrounds from the actual objects by considering deep semantic and geometric information. Its ability to perform precise segmentations even in challenging scenarios makes SAM particularly well-suited for analyzing technical components where accurate background-object separation is essential.

[0078] The segmentation algorithm, specifically SAM, is trained using a large initial training dataset 730. This training dataset 730 contains a large number of images of components 20, particularly of vehicles. Examples of components included in the training dataset 730 are: - Powertrain components such as gearbox housings, clutches, axles, exhaust pipes - Brake components such as brake discs, brake pads and brake lines - Cooling components such as coolant reservoirs - Chassis components such as shock absorbers - Engine parts such as cylinder heads, pistons, crankshafts and camshafts - Body components such as bumpers, doors and mudguards

[0079] The first training dataset 730 enables the segmentation algorithm to perform the segmentation of components 20 from the acquired image data 250 with high accuracy. The careful selection and annotation of the images in the first training dataset 730 ensures that the segmentation algorithm is able to accurately recognize and separate both general and vehicle-specific components.

[0080] Furthermore, the data processing module 400 includes an energy-based AI model 450. This AI model 450 is trained to perform pixel-based material identification. For this purpose, after segmenting the image data 250, a pixel or a group of pixels from the image data 250 of the captured image is selected and analyzed by the AI ​​model 450 to determine the material of the component 20.

[0081] The AI ​​model 450 uses an energy-based model with latent variables based on tactile measurements to predict the energy of a specific material and thus determine the probability that the sampled pixel belongs to that material. The latent variables represent, among other things, the roughness and frictional properties of the material, which are detected by biomimetic sensors 320.

[0082] The concept of an "energy-based model" is a specific approach within artificial intelligence and machine learning used for probability optimization and data modeling. It is based on the idea of ​​minimizing an "energy" that quantifies the accuracy or quality of a prediction or classification. Energy-based models (EBMs) use an energy function to evaluate the probability of a given classification or assignment. The goal is to minimize the energy required for correct assignments and maximize the energy required for incorrect assignments.

[0083] In this context, the term "energy" is an abstract mathematical quantity, i.e., a scalar value, and therefore not directly comprehensible in physical terms. It represents an evaluation of the "quality" of a particular solution or prediction by the AI ​​Model 450. The AI ​​Model 450 is designed to assign lower energy values ​​to correct or probable classifications and higher energy values ​​to incorrect or improbable classifications.

[0084] During the training process, the AI ​​model 450 attempts to adjust the parameters to minimize the energy required for correct predictions, thus increasing the likelihood and preference for these predictions. The energy function is a mathematical function that assigns an energy value to a given classification. The goal is to design this function in such a way that the energy required for correct classification is as low as possible.

[0085] Within the scope of the invention, the following architectures, which utilize latent variables and embeddings in different ways to make predictions, can be used as implementations of energy-based models (EBMs). a) Joint Embedding Predictive Architecture (JEPA): This architecture is designed to learn joint embeddings of input data. This means that the model attempts to find a representative representation for both the input data (e.g., an image) and the output (e.g., a material prediction) so that they are mapped in the same space. The proximity of these embeddings in the shared space is crucial for the prediction. JEPA can efficiently learn and utilize complex relationships between input and output data. This can be particularly advantageous when the data encompasses different modalities (e.g., visual and tactile information). b) Latent Variable Energy-Based Model (LVEBM): An LVEBM is an energy-based model that uses latent variables to model the relationship between input data and predictions. Latent variables are hidden, unobservable variables that the model learns during training. These variables capture complex patterns in the data that are crucial for prediction. LVEBM is able to model hidden relationships that are present in the data but not directly visible, leading to more accurate predictions, especially with low-resolution input data. c) Latent Variable Generative Energy-Based Model (LVGEBM): This architecture is an extension of LVEBM where the model focuses not only on prediction but also on learning generative aspects of the data. This means that the model is able to generate new data that resemble the original input data, based on the learned latent variables. LVGEBM can be used not only to analyze existing data, but also to generate synthetic data with properties similar to the real data. This can further increase the accuracy and flexibility of the model.

[0086] All three architectures utilize the fundamental principle of an energy-based model, where energy is minimized to achieve accurate predictions. Depending on the requirements of the specific application (e.g., whether the focus is on understanding latent variables or generating new data), one of these architectures can be selected to train the energy-based AI model 450.

[0087] The AI ​​model 450 attempts to correctly classify the material of component 20 based on one or more selected pixels. The energy function assigns an energy value to each possible material. The AI ​​model 450 is trained to minimize the energy associated with the correct material, thus favoring this material as the most likely prediction.

[0088] The AI ​​model 450 is trained using self-supervised learning (SSL) to utilize latent variables that represent important material properties. Latent variables are hidden or indirectly captured features that cannot be directly measured but are learned by the AI ​​model 450. They are crucial for capturing complex patterns and relationships in the data being evaluated that cannot be revealed through simple, direct observation. Within the scope of the present invention, the latent variables relate to roughness, friction properties, or other specific material properties that the AI ​​model 450 uses to make the correct material prediction. These latent variables help minimize energy consumption when the sampled pixel and the material image are a good match, indicating an accurate material prediction.

[0089] Database 700 contains an extensive collection of annotated images of components under various conditions, supplemented by tactile measurement data and specific information on the materials used. This data is used for both a first training dataset 730 for the image processing module 430 and a second training dataset 770 for the AI ​​model 450.

[0090] The annotations are of central importance for the training results of the AI ​​model 450 and contain a wealth of relevant details. These include material specifications such as the exact material type, e.g., aluminum or silicon steel, as well as specific alloy specifications and information about any surface treatments or heat treatments applied to component 20.

[0091] In addition, the annotations contain geometric properties such as the exact dimensions of component 20, its shape and structure, and its surface finish, for example, whether it is smooth, rough, or polished. Manufacturing information such as the manufacturing process used, manufacturer details, production data, and any serial numbers are also included in the annotations to ensure traceability and the specific production history of component 20.

[0092] Furthermore, the annotations provide insights into the functional role of component 20 within an assembly or system, its specific application area, and the typical operating conditions under which component 20 is used. Visual characteristics such as component 20's color, reflective properties, and textures are also documented to record its appearance under standardized lighting conditions.

[0093] Finally, the annotations also include information on the condition of component 20, including signs of wear, usage patterns, and an estimate of the remaining service life based on previous use and discernible wear. These comprehensive annotations enable the AI ​​model 450 to make more accurate predictions by considering not only visual data but also contextual information that is crucial for material identification.

[0094] Furthermore, database 700 also contains images of the components under various lighting conditions to enable the most comprehensive analysis possible. These images are annotated with descriptions of the lighting conditions, such as whether it is direct or diffuse light, which light source was used, and the angle from which the light falls on the component. This information is important because it helps to understand the material's reflective properties and improves the accuracy of material identification under different lighting conditions.

[0095] The training process utilizes the second training dataset, 770, which consists of component images, tactile measurement data, and annotations, to train and optimize the AI ​​model 450. The goal of the training is to maximize prediction accuracy by minimizing energy consumption. Energy is minimized when there is a high similarity between the sampled pixel and reference data, indicating a correct material prediction. Latent variables parameterize the complex relationships between the image pixels and the material properties, thus contributing to the improved performance of the AI ​​model 450.

[0096] Two different training methods can be used for training the energy-based AI model 450: the contrastive method and the regularized method.

[0097] Contrastive learning is a self-supervised learning technique in which the AI ​​model 450 learns to distinguish between similar and different examples. The contrastive method often uses "positive" and "negative" examples. Positive examples are pairs of images or data points that are considered similar or related. Two images of the same component from different angles can be positive examples. Negative examples are pairs of images or data points that are considered different or unrelated. Two images of different components or materials can be negative examples.

[0098] The contrastive method requires negative examples to teach the AI ​​model 450 which characteristics distinguish different materials. Without these negative examples, it can be difficult for the AI ​​model 450 to learn subtle differences between materials, as it lacks a reference point to recognize the distinction. Therefore, in the contrastive method, the AI ​​model 450 learns by comparing positive and negative examples, with the negative examples being crucial for learning the differences between the materials.

[0099] The regularized method is an approach to avoid overfitting by adding extra information or constraints during training. With the regularized method, the operating range of values ​​is defined. In this context, the "operating range" refers to the range of values ​​or states that the AI ​​model can assume during simulation or training. Unrealized values ​​refer to values ​​or states that could theoretically influence the behavior of the AI ​​model 450 but are not explicitly included in the training data.

[0100] The operating range of these values ​​is used to regularize the latent variables. This means that constraints or additional criteria are applied to the latent variables to ensure they remain within a specific range or meet certain criteria. This makes the AI ​​model 450 more stable and less prone to overfitting, as it learns to recognize meaningful and generalizable patterns without being overly reliant on specific training data.

[0101] These methods offer various approaches to improving model performance and handling latent variables in an energy-based AI model 450.

[0102] After the acquisition of image data 250, which provides a representation of component 20, and the acquisition of tactile sensor data 350 of component 20 by the biomimetic sensor 320, the user interface 500 is used as a means of communication for interaction with the trained AI model 450. The user interface 500 serves as an interface between the user and the AI ​​model 450 and allows the user to submit queries for material determination of component 20. It can be designed as an interactive user interface that runs on various devices such as a touchscreen, a desktop computer, or a mobile device.

[0103] The user can enter a prompt 550 in the form of a text message or a voice message via the user interface 500. For example, a user can enter a text message such as "Find the exhaust manifold material" on a touchscreen or record a voice message via a microphone as a prompt 550. This prompt 550 is then forwarded to a speech model 470 for voice message processing.

[0104] The Language Model 470 uses speech recognition technology to convert the recorded voice message into text. It analyzes the text of the message or voice message to extract the essential information and context of the prompt 550. This ensures that the prompt 550 is presented in a form that can be interpreted by the AI ​​Model 450.

[0105] After the input prompt 550 is interpreted by the language model 470 and transmitted to the AI ​​model 450, the AI ​​model 450 begins analyzing the captured image data 250, which contains a representation of component 20, as well as the tactile sensor data 350 of component 20. In the first step, the AI ​​model 450 segments the image representation using the image processing module 430 to precisely isolate the relevant component 20. In the second step, the AI ​​model 450 uses the previously trained algorithms to analyze the material properties of the identified component 20 pixel by pixel. The features extracted from the tactile sensor data 350 and the visually captured data are evaluated in combination. The AI ​​model 450 compares these material properties with the reference data stored in the database 700 and determines the most probable material.

[0106] After the AI ​​model 450 has analyzed and determined the material of component 20, the analysis result 570 is displayed via the user interface 500. The user receives clear and understandable feedback about the identified material in the form of a text message or a visual representation. The text message can contain additional contextual information provided by the annotation of the images in the first training dataset 730 and / or the second training dataset 770. This additional information can include details about the specific application of the material, typical areas of use, or relevant properties and processing methods, providing the user with further important information.

[0107] In Fig. Figure 2 describes the process steps for the automated determination of material properties of a component 20, in particular for a vehicle.

[0108] In step S10, image data 250 of one or more image representations of the component 20 are acquired with an image acquisition device 220.

[0109] In step S20, tactile sensor data 350 of component 20 are acquired with a biomimetic sensor 320.

[0110] In step S30, an input prompt 550 for material determination is entered into a user interface 500.

[0111] In step S40, the input prompt 550 is processed by a language model 470.

[0112] In step S50, the component 20 is segmented from the image data 250 of the recorded image representation of the component 20 using an image processing module 430.

[0113] In step S60, the material of component 20 is analyzed and determined using an AI model 450 that processes the image data 250 and the tactile sensor data 350 in combination.

[0114] In step S70, the analysis results 570 of the material determination are displayed on the user interface 500 in the form of a text message or a visual representation.

[0115] Fig. Figure 3 schematically represents a computer program product 900 comprising an executable program code 950 configured to perform the method according to the second aspect of the present invention.

[0116] The system according to the invention offers flexible and precise material identification through the integration of image acquisition and tactile measurement of a component. It allows the use of various image acquisition devices, such as conventional cameras or VR glasses, which can be selected depending on the specific task. The integration of tactile measurements results in significantly improved material recognition, far exceeding the capabilities of purely visual methods. The energy-based model is characterized by high accuracy in material identification, as it has been trained on a large number of components under varying conditions and with different component geometries. The combination of these technologies enables a reliable and accurate solution for material identification in diverse application areas.

[0117] The invention can be used, for example, for quality control in production to ensure that the correct materials have been used. It is also useful in the maintenance and inspection of components, particularly in hard-to-reach areas, for carrying out precise material analyses. In the recycling sector, the invention helps to sort waste materials by enabling accurate material identification.

Claims

[1] System (100) for the automated determination of material properties of a component (20), in particular for a vehicle, comprising an image acquisition module (200) with at least one image acquisition device (220) configured to acquire image data (250) of one or more image representations of the component (20) under different conditions, wherein the component (20) may be arranged in a larger assembly; a sensor module (300) with at least one biomimetic sensor (320) that acquires tactile sensor data (350) of the component (20), including roughness and friction properties of the at least one material of the component (20);a data processing module (400) comprising an image processing module (430), an AI model (450) and a language model (470), wherein the image processing module (430) is trained to perform a segmentation of the component (20) on the image representation based on the recorded image data (250), wherein the AI ​​model (450) comprises an energy-based model and is trained to analyze and determine the material properties of the component (20) based on the image data (250) and the sensor data (350), wherein the language model (470) is trained to interpret input prompts (550) in the form of text or voice messages and forward them to the AI ​​model (450); and a user interface (500) designed to capture the input prompts (550) and forward them to the data processing module (400), and to display the analysis result (570) of the material determination as a text message or visual representation. [2] System (100) according to claim 1, wherein the image acquisition device (220) is configured as a high-resolution camera, smartphone or VR glasses to capture 2D representations or 3D representations or immersive 3D representations of the component (20) in the form of image data (250). [3] System (100) according to claim 1 or 2, wherein the AI ​​model (450) is implemented as a Joint Embedding Predictive Architecture (JEPA), Latent Variable Energy-Based Model (LVEBM) or Latent Variable Generative Energy-Based Model (LVGEBM). [4] System (100) according to one of the preceding claims, wherein the biomimetic sensor (320) is configured to detect stick-slip behavior (stick-slip sensing), and wherein the acquired sensor data (350) provide information about the friction properties of the material and are used to determine latent variables that are crucial for the correct assignment of the material by the AI ​​model (450). [5] System (100) according to one of the preceding claims, wherein the AI ​​model (450) was trained with a second training data set (770) which comprises annotated images of components and measurement results of tactile measurements of different materials stored in a database (700), wherein the annotations in particular contain information about specific material properties and / or different lighting situations, and wherein during the training of the AI ​​model (450) the latent variables based on the tactile measurements were learned by the AI ​​model (450) to correctly assign the correct material. [6] System (100) according to one of the preceding claims, wherein the image processing module (430) uses a segmentation algorithm, which is implemented in particular by the “Segment Anything Model” (SAM), and wherein the segmentation algorithm was trained with a first training data set (730) comprising a plurality of images of components (20), in particular of vehicles, especially images of powertrain components such as gearbox housings, clutches, axles and exhaust pipes, brake components such as brake discs, brake pads and brake lines, cooling components such as coolant reservoirs, chassis components such as shock absorbers, engine components such as cylinder heads, pistons, crankshafts and camshafts, and body components such as bumpers, doors and fenders. [7] System (100) according to any of the preceding claims, wherein the user interface (500) comprises a touchscreen, a desktop computer or a mobile device to enable interactive user interaction. [8] Method for the automated determination of material properties of a component (20), in particular for a vehicle, comprising the following process steps: - Acquisition (S10) of image data (250) of one or more image representations of the component (20) using an image acquisition device (220); - Acquisition (S20) of tactile sensor data (350) of the component (20) with a biomimetic sensor (320), wherein the tactile sensor data (350) are acquired by a biomimetic sensor (320) designed to detect stick-slip behavior (stick-slip sensing), and wherein the acquired sensor data (350) provide information about the friction properties of the material and are used to determine latent variables that are crucial for the correct assignment of the material by an AI model (450); - Entering (S30) a prompt (550) for material determination into a user interface (500); - Processing (S40) the input prompt (550) by a language model (470); - Segmenting (S50) of the component (20) from the image data (250) of the recorded image representation of the component (20) using an image processing module (430); - Analyzing (S60) and determining the material of the component (20) using an AI model (450) that processes the image data (250) and the tactile sensor data (350) in combination, wherein the material determination is carried out by an energy-based model of the AI ​​model (450) that is implemented as a Joint Embedding Predictive Architecture (JEPA) or Latent Variable Energy-Based Model (LVEBM); - Display (S70) the analysis result (570) of the material determination on the user interface (500) in the form of a text message or a visual representation. [9] Method according to claim 8, wherein the image capture is performed using a high-resolution camera, a smartphone or VR glasses to capture 2D representations or 3D representations or immersive 3D representations of the component (20) in the form of image data (250). [10] Method according to claim 8 or 9, wherein the AI ​​model (450) is trained with a second training data set (770) which comprises annotated images of components and measurement results of tactile measurements of different materials stored in a database (700), wherein the annotations in particular contain information about specific material properties and / or different lighting situations, and wherein during the training of the AI ​​model (450) latent variables based on the tactile measurements are learned by the AI ​​model (450) for the subsequent correct assignment of the correct material. [11] Method according to any one of claims 8 to 10, wherein the image processing module (430) uses a segmentation algorithm, which is implemented in particular by the “Segment Anything Model” (SAM), and wherein the segmentation algorithm is trained with a first training data set (730) comprising a plurality of images of components (20), in particular of vehicles, especially images of powertrain components such as gearbox housings, clutches, axles and exhaust pipes, brake components such as brake discs, brake pads and brake lines, cooling components such as coolant reservoirs, chassis components such as shock absorbers, engine components such as cylinder heads, pistons, crankshafts and camshafts, and body components such as bumpers, doors and fenders. [12] Method according to one of claims 8 to 11, wherein the input prompt (550) for material determination of the component (20) is in the form of a text or voice message and is processed by the language model (470), and wherein the display of the analysis result (570) of the material determination includes additional context information taken from the annotations of the images used in the first training data set (730) or second training data set (770). [13] Computer program product (900) comprising an executable program code (950) configured to perform the method according to any one of claims 8 to 12 when executed.

Images