SYSTEMS AND METHODS FOR TESTING A VEHICLE

Abstract

A method includes receiving a video stream from a mixed-media device that captures one or more components of a vehicle, detecting one or more defects associated with the one or more components of the vehicle, transmitting dynamic feedback to an operator of the mixed-media device from a processor, and guiding the operator to deal with the one or more defects via the mixed-media device.

Description

AREA

[0001] The present disclosure relates to the testing of one or more components of a vehicle and in particular to systems and methods for vehicle testing using a mixed-media device. GENERAL STATE OF THE ART

[0002] The statements in this section provide only background information regarding the present disclosure and may not represent the state of the art. Warranty and in-house repair problems associated with vehicle production are typically due to improperly installed components rather than defective components. Since verifying whether a component has been properly installed can be considered an additional manufacturing-related task, it may be necessary to re-evaluate the production process to accommodate such verification. However, without real-time verification of component installation, inefficiencies in the production process may arise, such as waste, rework, unsatisfactory communication, or a combination thereof.

[0003] This disclosure addresses these and other problems relating to the testing of the vehicle. SUMMARY

[0004] This section provides a general summary of the revelation and is not a comprehensive revelation of its full scope or all of its features.

[0005] The present disclosure provides a method comprising: receiving, by a processor, a video stream from a mixed-media device that captures one or more components of a vehicle; detecting, by an algorithm associated with the processor, one or more defects associated with the one or more components of the vehicle in response to the receipt of the video stream; transmitting, via the mixed-media device, dynamic feedback from the processor to an operator of the mixed-media device based on the detection of the one or more defects; and instructing the operator, via the mixed-media device, to address the one or more defects based on the transmission of the dynamic feedback; wherein the video stream comprises one or more images within a field of view of the mixed-media device;wherein the dynamic feedback includes one or more instructions, the method further comprising: transmitting the one or more instructions to the operator via the mixed-media device, wherein the one or more instructions facilitate the control of the field of view of the mixed-media device in order to successfully complete an inspection of the one or more components; wherein the dynamic feedback is displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more transducers of the mixed-media device, or a combination thereof;wherein the transmission of dynamic feedback to the operator further includes: causing the operator to be shown a dynamic update of the user interface, wherein the dynamic update indicates a proper installation of the one or more components via one or more bounding boxes associated with the one or more components, a color associated with the proper installation of the one or more components, a color change indicating that the operator has dealt with the one or more defects, or a combination thereof;wherein the dynamic feedback includes one or more augmented reality images, the method further comprising: generating the one or more augmented reality images based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions specifying the proper installation of the one or more components; and displaying the one or more augmented reality images on the user interface of the mixed-media device;wherein the dynamic feedback includes a first set of one or more video instructions or a second set of one or more video instructions, the method further comprising: transmitting, via the mixed-media device, the first set of one or more video instructions to the operator based on the captured one or more components, wherein the first set of one or more video instructions includes one or more steps for the proper installation of the one or more components;or transmit, via the mixed-media device, the second set of one or more video instructions to the operator based on the detection of one or more defects, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and the one or more steps for properly installing the one or more components; and wherein the dynamic feedback includes one or more voice notes, the method further comprising: receiving, by the processor, one or more voice notes based on a voice command from the operator, wherein the one or more voice notes are associated with the video stream from the mixed-media device, and wherein the voice command specifies one or more defects detected by an operator.

[0006] The present disclosure provides a system comprising: a processor configured to: receive a video stream from a mixed-media device capturing one or more components of a vehicle; detect, by means of an algorithm associated with the processor, one or more defects associated with the one or more components of the vehicle in response to receiving the video stream; transmit, via the mixed-media device, dynamic feedback to an operator of the mixed-media device from the processor based on the detection of the one or more defects; and instruct the operator, via the mixed-media device, to address the one or more defects based on the transmission of the dynamic feedback.and the mixed-media device configured to: transmit the video stream, receive the dynamic feedback, and communicate the dynamic feedback to the operator, wherein the dynamic feedback is displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more transducers of the mixed-media device, or a combination thereof; wherein the video stream comprises one or more images within a field of view of the mixed-media device; wherein the dynamic feedback includes one or more instructions, and the processor is further configured to: transmit the one or more instructions to the operator via the mixed-media device, wherein the one or more instructions facilitate the control of the field of view of the mixed-media device to successfully complete an inspection of the one or more components;wherein the processor, which is configured to transmit dynamic feedback to the operator, is further configured to: cause the operator to be shown a dynamic update of the user interface, wherein the dynamic update indicates a proper installation of the one or more components via one or more bounding boxes associated with the one or more components, a color associated with the proper installation of the one or more components, a color change indicating that the operator has dealt with the one or more defects, or a combination thereof;wherein the dynamic feedback includes one or more augmented reality images, the processor is further configured to: generate the one or more augmented reality images based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions specifying the proper installation of the one or more components; and display the one or more augmented reality images on the user interface of the mixed-media device;wherein the dynamic feedback includes a first set of one or more video instructions or a second set of one or more video instructions, wherein the processor is further configured to: transmit, via the mixed-media device, the first set of one or more video instructions to the operator based on the detected one or more components, wherein the first set of one or more video instructions includes one or more steps for the proper installation of the one or more components;or transmit, via the mixed-media device, the second set of one or more video instructions to the operator based on the detection of one or more defects, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and the one or more steps for properly installing the one or more components; and wherein the dynamic feedback includes one or more voice notes, the processor further being configured to: receive, through the processor, one or more voice notes based on a voice command from the operator, the one or more voice notes being associated with the video stream from the mixed-media device.

[0007] The present disclosure provides one or more non-transitory, computer-readable media that store processor-executable instructions which, when executed by at least one processor, cause the at least one processor to: receive, by a processor, a video stream from a mixed-media device that captures one or more components of a vehicle; detect, by an algorithm associated with the processor, one or more defects associated with the one or more components of the vehicle in response to receiving the video stream; transmit, via the mixed-media device, dynamic feedback to an operator of the mixed-media device from the processor based on the detection of the one or more defects; and instruct the operator, via the mixed-media device, to address the one or more defects based on the transmission of the dynamic feedback.wherein the video stream comprises one or more images within a field of view of the mixed-media device; wherein dynamic feedback includes one or more instructions; and wherein the at least one processor is further caused to: transmit the one or more instructions to the operator via the mixed-media device, wherein the one or more instructions facilitate steering the field of view of the mixed-media device to successfully complete an inspection of the one or more components; wherein the dynamic feedback is displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more transducers of the mixed-media device, or a combination thereof;and wherein the at least one processor that is caused to transmit the dynamic feedback to the operator is further caused to: cause the operator to be shown a dynamic update of the user interface, wherein the dynamic update indicates a proper installation of the one or more components via one or more bounding boxes associated with the one or more components, a color associated with the proper installation of the one or more components, a color change indicating that the operator has dealt with the one or more defects, or a combination thereof.

[0008] Further areas of application will become apparent from the description provided in this document. It is understood that the description and specific examples serve only for illustration and are not intended to limit the scope of this disclosure. DRAWINGS

[0009] For a comprehensive understanding of the revelation, various forms of it will now be described by way of example with reference to the accompanying drawings, in which the following applies: Fig. 1 is a block diagram of a vision system associated with the inspection of one or more components of a vehicle according to one or more embodiments of the present disclosure; Fig. 2 is an exemplary environment with respect to the vision system according to one or more embodiments of the present disclosure; Fig. 3 is a flowchart illustrating an exemplary procedure for carrying out the testing of one or more components associated with a vehicle according to one or more embodiments of the present disclosure; Fig. Figure 4 is a flowchart illustrating a further exemplary method for carrying out the testing of one or more components associated with a vehicle according to one or more embodiments of the present disclosure; and Fig. Figure 5 is a block diagram illustrating an exemplary computer system according to one or more embodiments of the present disclosure.

[0010] The drawings described in this document serve only for illustration and are not intended to limit the scope of the present disclosure in any way. DETAILED DESCRIPTION

[0011] The following description is merely exemplary and is not intended to limit the present disclosure, application, or uses. It is understood that in all drawings, corresponding reference numerals indicate identical or corresponding parts and features.

[0012] One or more of the examples described in this document provide systems and procedures for vehicle inspection that can reduce the tasks performed by an operator in real time and allow for a more optimized process (e.g., correct angle, adequate lighting, sufficient proximity to a vehicle, etc.). For example, real-time vehicle inspection can be performed within a production process. The integration of real-time video collection for the operator (e.g., shortly before installation), which may include providing an improved viewing angle for the image collection, can also be provided. Additionally, real-time streaming of the video and / or images via Wi-Fi, 5G Bluetooth®, or other means to a computing device can be provided to draw conclusions from the video and / or images.

[0013] In one or more examples, the video and / or images may originate from one or more cameras on a mixed-media device. Additionally, the mixed-media device's screen may provide the operator with feedback regarding checks that should be completed, as well as indicating the check result (e.g., success or otherwise) and when a check was completed. In one or more examples, the operator may be provided with guidance directing them to where to look to perform the next task and / or check (e.g., audibly and / or visually).Furthermore, the operator can be provided with immediate feedback on the unsuccessful test, allowing the operator to correct the error, use one or more voice commands to mark the problem(s) for subsequent repair in the production process, and / or use one or more voice commands to provide feedback to the system that the test was unsuccessful.

[0014] One or more examples provide the ability to override a test, which may cause the system to save the source video and forward it to an artificial intelligence training platform for additional model training. One or more photos of potential issues can also be sent to another party and combined with annotations in the form of feedback. One or more still images of unsuccessful tests can be captured and, in one or more examples, routed to downstream workstation(s) for repair, along with an option for the operator to add one or more voice notes so that the repair technician can take various actions (e.g., calling a specific person for details related to the test).

[0015] One or more examples may additionally provide the ability to use speech tools to gather operator feedback; continuous improvement of ideas; provision of one or more alerts to an operator; recording of video(s) for operator training; the ability to collect training video(s) during the production process; the ability to send a message to the operator to alert them to an approaching vehicle; provision of a model mix; provision of an operator task list and / or an indication of when a production process action has been taken (e.g., completed, bypassed, etc.); provision of guidance during an event (e.g., natural disaster, active shooter, etc.); or integration of one or more learning models.

[0016] Fig. Figure 1 illustrates an exemplary vision system 100 that is assigned to an inspection of one or more components belonging to a vehicle (e.g., a vehicle 214, as in Fig. 2 shown). However, it is understood that the vision system 100 can be applied to a test process assigned to any structure and is not limited to the testing of the vehicle 214 and / or the one or more components assigned to the vehicle 214.

[0017] In one or more embodiments, the testing of one or more components of the vehicle 214 is carried out by a mixed-media device (e.g., a mixed-media device 200, as in Fig. (2 shown) in communication with a defect detection processor 102. The defect detection processor 102 is configured to detect one or more defects associated with one or more components of the vehicle 214 in response to receiving a video stream (e.g., a real-time or live video stream) originating from the mixed-media device 200. For example, the mixed-media device 200 can be any electronic device wearable by an operator (e.g., a human operator), such as, but not limited to, eyeglasses, a head-mounted camera, or a body camera. It is understood that the defect detection processor 102 can be located partially or completely externally within a computer (e.g., an edge PC) separate from the mixed-media device 200, or it can be located partially or completely within the mixed-media device 200.

[0018] With reference to Fig. 2. The mixed-media device 200 can include one or more lenses 202, one or more sensors 204, one or more covers 206, and one or more brackets 208. It is understood that, while the mixed-media device 200, which is in Fig. As shown in Figure 2, which depicts a duality of lenses 202, a duality of sensors 204, a duality of covers 206 (e.g., housings), and a duality of brackets 208, the mixed-media device 200 can include any number (e.g., more or less than two) of lenses 202, sensors 204, covers 206, and / or brackets 208. In one or more embodiments, the one or more lenses 202 are configured to display content to the operator of the mixed-media device 200. However, it is understood that the operator of the mixed-media device 200 is also able to view any environment within a field of view 216 of the operator beyond the one or more lenses 202 by viewing the environment through the one or more lenses 202.In other words, the display of the content does not impede the operator's ability to view the surroundings beyond the one or more lenses 202 that would otherwise be within the operator's field of vision 216 of the mixed-media device 200 (e.g., viewed through the lenses 202).

[0019] The one or more sensors 204 are arranged on the mixed-media device 200 near the one or more lenses 202 and are configured to capture the operator's field of view 216. However, it is understood that the one or more sensors 204 may be arranged on or integrated into any section of the structure associated with the mixed-media device 200. The one or more sensors 204 may have one or more camera, lidar, radar, and / or ultrasonic capabilities.

[0020] Additionally, the one or more enclosures 206 can include one or more transducers 210 and / or one or more receivers 212. For example, the one or more transducers 210 can be an array of loudspeakers configured to output audio sound waves to the operator of the mixed-media device 200. Alternatively, the one or more receivers 212 can be an array of microphones configured to receive one or more voice commands (e.g., one or more voice memos) from the operator. For example, the one or more voice commands can include feedback observed by the operator regarding the one or more components, providing contextual feedback to the fault detection processor 102.As another example, contextual feedback can improve the processing of an input 104 by the predictive algorithm 106, as described in this document. The one or more panels 206 are attached to the one or more brackets 208 of the mixed-media device 200. However, it is understood that the one or more panels 206 can be attached to any section of the construct associated with the mixed-media device 200.

[0021] With renewed reference to Fig. 1. The video stream and / or one or more voice commands are received as input 104 by the defect detection processor 102. For example, input 104 is transmitted wirelessly from the mixed-media device 200 to the defect detection processor 102. Alternatively, the transmission of input 104 is supported wirelessly by a wireless communication protocol (e.g., a protocol such as Bluetooth). ®, a mobile communication protocol, a wireless fidelity (Wi-Fi) protocol, a near-field communication (NFC) protocol, an ultra-wideband (UWB) protocol, or a light fidelity (Li-Fi) protocol). As an example, the defect detection processor 102 is also configured to receive one or more voice commands as input 104. The defect detection processor 102 includes a predictive algorithm 106 configured to process the received video stream and / or voice commands (e.g., input 104).

[0022] As an example, and based on the processing of input 104, the predictive algorithm 106 is configured to assemble each image from a multitude of individual images if input 104 is received as a multitude of individual images. As another example, the dynamically received input 104 allows for training (e.g., supervised or unsupervised) of the predictive algorithm 106. As a further example, the training of the predictive algorithm 106 can be improved based on context feedback received as one or more voice commands. As yet another example, and based on the processing of input 104, the predictive algorithm 106 is configured to analyze the input 104 to detect any defects associated with any one or more components.As another example, and based on the processing of input 104 and / or the detection of any defects associated with any of the one or more components, the predictive algorithm 106 can provide the operator of the mixed-media device 200 with optimized feedback by exchanging one or more data elements (associated, for example, with input 104) with one or more machine learning models 108. The one or more machine learning models 108 may include a bounding frame model 110, an augmented reality model 112, and a direction model 114. It is understood, however, that the one or more machine learning models 108 may include any number of models with respect to the inspection of the vehicle 214.

[0023] In one or more embodiments, the predictive algorithm 106 is configured to process the input 104 to determine whether one or more components are improperly installed with respect to the vehicle 214. In response to the predictive algorithm 106 determining that one or more components are improperly installed with respect to the vehicle 214, the bounding box model 110 is configured to generate a bounding box (e.g., a virtual bounding box) around the one or more improperly installed component(s), which is displayed by the mixed-media device 200, as described in more detail in this document.It is understood, however, that bounding box model 110 can generate a bounding box around each of the one or more components, regardless of whether the one or more components are improperly installed. In a case where bounding box model 110 generates a bounding box around each of the one or more components, the bounding box model 110 can highlight the one or more improperly installed component(s) by causing the bounding box around the one or more improperly installed component(s) to appear as a highlighted or marked frame, such as thicker, lighter, and / or a different color than the bounding boxes generated around each of the one or more properly installed components (e.g., displayed to the operator).It is understood that the limiting frame model 110 can highlight the one or more improperly installed component(s) in any way, whereby the operator is able to distinguish the one or more improperly installed component(s) from the one or more properly installed components.

[0024] In one or more embodiments, and in a case where any one or more components have one or more potential mating orientations, the Augmented Reality Model 112 is configured to generate feedback associated with the proper installation (e.g., proper alignment) of the one or more components exhibiting the one or more potential mating orientations. Likewise, in a case where any one or more components can be placed into one or more modes, the Augmented Reality Model 112 is configured to generate feedback associated with the proper installation of the one or more components exhibiting the one or more potentially practicable modes.It is understood that in any case the augmented reality model 112 can generate the feedback associated with the proper installation of one or more components in response to the processing of the input 104 by the predictive algorithm 106 (e.g. based on the operator's field of vision 216 or one or more voice commands from the operator).

[0025] In one or more embodiments, the predictive algorithm 106 is configured to process the input 104 to determine whether one or more components are improperly installed with respect to the vehicle 214. In response to the predictive algorithm 106 determining that one or more components are improperly installed with respect to the vehicle 214, the direction model 114 is configured to display one or more instructions to the operator of the mixed-media device 200. For example, the one or more instructions may include a procedure for removing the improperly installed component(s) and a procedure for properly installing the one or more components.It is understood, however, that the direction model 114 is configured to display one or more instructions to the operator regardless of whether a determination is made as to whether one or more components relating to the vehicle 214 are not properly installed. For example, the direction model 114 may be configured to display one or more instructions to the operator in response to the processing of input 104 by the predictive algorithm 106 (e.g., based on the operator's field of vision 216 or one or more voice commands from the operator). It is also understood that the one or more instructions may be step-by-step instructions, such as video instructions, and / or a displayed list of instructions.It is further understood that the one or more instructions can be any form of instructions relating to the procedure and associated with the installation of the one or more components.

[0026] As another example, the one or more instructions may include one or more directional indicators (e.g., directional arrows) that instruct the operator to change the field of view 216 associated with the environment viewed through the one or more lenses 202. For example, the one or more directional indicators may instruct the operator to view a different component than the one or more components originally viewed. As yet another example, the one or more directional indicators may be generated in a case where the inspection of the one or more components requires multiple angles, images, or components that cannot be viewed within a single field of view 216.In other words, the one or more directional indicators can be generated in a case where the inspection of the one or more components requires the operator to move around the vehicle 214 (e.g., change their position) to capture every required viewing angle of each of the one or more components being inspected. It is understood that the one or more directional indicators can be provided to the operator based on a global reference point of any one or more components or a local reference point associated with the current position of the mixed-media device 200 relative to any one or more components. The one or more directional indicators can be any directional indicator and can be of any type and take on different forms and configurations.

[0027] As another example, the one or more instructions may also include one or more video-based instructions showing the location of any one or more components, as well as an example of the proper installation of the one or more components shown. As yet another example, the one or more instructions may also include one or more audio-based instructions provided to the operator of the mixed-media device 200 via the one or more transducers 210. As a further example, the one or more instructions may fix any one or more components and guide the operator to adjust the field of view 216 based on tracking the fixed one or more components.

[0028] In one or more embodiments, the defect detection processor 102 is configured to provide feedback to the operator of the mixed-media device 200 as output 116. For example, the feedback can be displayed visually on one or more lenses 202 of the mixed-media device 200 and / or audibly provided to the operator via one or more transducers 210 of the mixed-media device 200. For example, the bounding box generated around one or more improperly installed components is displayed to the operator via the mixed-media device 200. As another example, the feedback generated by the augmented reality model 112 is displayed to the operator via the mixed-media device 200. As yet another example, the one or more instructions are displayed and / or audibly provided to the operator via the mixed-media device 200.In any case, it is understood that output 116 instructs the operator of the mixed-media device 200 on how to handle (e.g., repair, mitigate, etc.) an instance of one or more defects associated with one or more components of the vehicle 214. In one or more embodiments, the operator of the mixed-media device 200 may receive a prioritized list of the order in which the one or more defects should be handled based on one or more considerations, such as the severity of the one or more defects, the speed at which the one or more defects should be repaired, or the proximity of the one or more defects to the operator. For example, the prioritized list may be generated by the predictive algorithm 106.

[0029] Fig. Figure 3 is a flowchart illustrating an exemplary procedure 300 for checking one or more components associated with a vehicle (e.g., vehicle 214) within a vision system (e.g., vision system 100). In procedure 302, a video stream (e.g., input 104) is received from a mixed-media device (e.g., mixed-media device 200). For example, the video stream is received at a processor (e.g., the defect detection processor 102). As another example, the video stream captures one or more components of the vehicle. As yet another example, the video stream includes one or more images within a field of view (e.g., field of view 216) of the mixed-media device.

[0030] In process 304, one or more defects are detected that are associated with one or more components of the vehicle. For example, the one or more defects are detected by an algorithm (e.g., predictive algorithm 106) assigned to the processor. As another example, the one or more defects are detected in response to receiving the video stream.

[0031] In process 306, dynamic feedback (e.g., output 116) is transmitted from the processor to the operator of the mixed-media device. For example, the dynamic feedback is transmitted to the operator of the mixed-media device via the mixed-media device. In another example, the dynamic feedback is transmitted based on the detection of one or more defects. As yet another example, the dynamic feedback is displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more transducers (e.g., one or more transducers 210) of the mixed-media device, or a combination thereof. The user interface can be a dynamic display and / or a communication method by which the operator can communicate, for example, with the defect detection processor 102 via the mixed-media device 200.

[0032] As another example, the user interface is provided to the operator via one or more lenses (e.g., the one or more lenses 202) of the mixed-media device. As a further example, the dynamic feedback includes one or more instructions. In one or more embodiments, the one or more instructions are transmitted to the operator. For example, the one or more instructions are transmitted to the operator via the mixed-media device. As another example, the one or more instructions facilitate the control of the mixed-media device's field of view to successfully complete an inspection of the one or more components.

[0033] In yet another example, the dynamic feedback includes one or more augmented reality images. In another embodiment, the one or more augmented reality images are generated based on the one or more captured components. In yet another embodiment, the one or more augmented reality images are displayed on the user interface of the mixed-media device. For example, the one or more augmented reality images include one or more instructions specifying the proper installation of the one or more components.

[0034] As another example, the dynamic feedback includes an initial set of one or more video instructions. In one or more other embodiments, the initial set of one or more video instructions is transmitted to the operator. For example, the initial set of one or more video instructions is transmitted to the operator via the mixed-media device. As another example, the transmission of the initial set of one or more video instructions is based on the one or more captured components. As yet another example, the initial set of one or more video instructions includes one or more steps for the proper installation of the one or more components.

[0035] As another example, the dynamic feedback includes a second set of one or more video instructions. In one or more further embodiments, the second set of one or more video instructions is transmitted to the operator. For example, the second set of one or more video instructions is transmitted to the operator via the mixed-media device. As another example, the transmission of the second set of one or more video instructions is based on the detection of one or more defects. As yet another example, the second set of one or more video instructions includes one or more steps for removing one or more components from the vehicle and / or one or more steps for properly installing one or more components.

[0036] In yet another example, the dynamic feedback includes one or more voice memos. In yet another embodiment, the one or more voice memos are received by the processor. For example, the one or more voice memos are received based on a voice command from the operator. As another example, the one or more voice memos are associated with the video stream from the mixed-media device. As yet another example, the voice command indicates one or more defects detected by an operator.

[0037] In one or more embodiments, a dynamic update of the user interface is displayed to the operator. For example, the dynamic update indicates proper installation of one or more components via one or more bounding boxes associated with the one or more components, a color associated with proper installation of the one or more components, a color change indicating that the operator has addressed the one or more defects, or a combination thereof. In Operation 308, the operator is guided to address the one or more defects based on the transmission of the dynamic feedback.

[0038] Fig. Figure 4 is a flowchart illustrating another exemplary procedure 400 for checking one or more components associated with a vehicle (e.g., vehicle 214) within a vision system (e.g., vision system 100). In procedure 402, a video stream (e.g., input 104) is received by a mixed-media device (e.g., mixed-media device 200). For example, the video stream is received by a processor (e.g., defect detection processor 102). As another example, the video stream captures one or more components of the vehicle.

[0039] In process 404, the processor performs an analysis to determine if one or more defects are present that are associated with one or more vehicle components. If no defects are found that are associated with one or more vehicle components, the analysis is repeated (e.g., at a later time, after receiving another input 104, etc.). However, if one or more defects associated with one or more vehicle components are detected, in process 406, the processor transmits dynamic feedback (e.g., output 116) to the mixed-media device operator. For example, the dynamic feedback is transmitted via the mixed-media device to the mixed-media device operator.For example, the dynamic feedback transmitted in operation 408 (“Output 1”) includes one or more instructions that facilitate directing the field of view of the mixed-media device to successfully complete an inspection of one or more components. As another example, the dynamic feedback transmitted in operation 410 (“Output 2”) includes one or more augmented reality images generated based on the one or more captured components and then displayed on a user interface of the mixed-media device. For example, the one or more augmented reality images include one or more instructions indicating the proper installation of the one or more components.

[0040] As another example, the dynamic feedback transmitted in Operation 412 (“Output 3”) includes an initial set of one or more video instructions containing one or more video-based steps for the proper installation of the one or more components. As yet another example, the dynamic feedback transmitted in Operation 414 (“Output 4”) includes a second set of one or more video instructions containing one or more video-based steps for removing the one or more components from the vehicle and / or the one or more steps for the proper installation of the one or more components. As a further example, the dynamic feedback transmitted in Operation 416 (“Operation 5”) includes one or more voice notes received based on a voice command from the operator, indicating one or more defects identified by the operator.In process 418, the operator is instructed to handle the one or more defects based on the transmission of dynamic feedback in processes 408-416.

[0041] Fig. Figure 5 illustrates an operating environment that facilitates the implementation of one or more of the systems and procedures described in this document. In particular, the systems and procedures described in this document may be implemented using a computing device 502. For example, the computing device 502 may be a personal computer, a desktop computer, a laptop computer, a tablet, a handheld computer, a server, a workstation, a mainframe computer, a portable computer, a supercomputer, or a combination thereof. It is understood, however, that the foregoing examples of computing device 502 are not exhaustive and that the computing device 502 may be any type of processing or computing device.The computing device 502 generally includes a processor 504, a display adapter 506, one or more input / output ports 508, one or more input / output components 510, a network adapter 512, a power supply 514, and a memory 516. However, it is understood that the computing device 502 may include any additional components and need not include any of the listed components (e.g., the processor 504, the display adapter 506, the one or more input / output ports 508, the one or more input / output components 510, the network adapter 512, the power supply 514, and the memory 516).

[0042] The processor 504 is configured to provide instructions to the computing device 502, enabling the computing device 502 to process one or more tasks, including the execution of a software program to perform one or more operations, as described in more detail in this document. It is also understood that the computing device 502 can contain any number of processors 504. The display adapter 506 can be a graphics card or a video board that provides the computing device 502 with the ability to display content on a display device 518.For example, the display device 518 can be any screen, monitor, and / or light-emitting component associated with any personal computer, desktop, laptop, tablet, handheld computer, server, workstation, mainframe, portable computer, supercomputer, or a combination thereof. In some examples, the display device 502 forms part of the mixed-media device 200 and is configured to display a user interface. However, it is understood that the foregoing examples of the display device 518 are not exhaustive and that the display device 518 can be any type of device capable of providing a visual display.

[0043] The input / output port(s) 508 provides a number of interfaces (e.g., sockets) for one or more cables to be connected to the computing device 502. It is understood that any number of input / output ports 508 may be present on the computing device 502. For example, the input / output port(s) 508 provides a means for the computing device 502 to receive signals and / or data from an external device connected to the computing device 502 by one or more cables. As another example, the input / output port(s) 508 provides a means for the computing device 502 to send signals and / or data to an external device connected to the computing device 502 by one or more cables.The input / output component(s) 510 may include one or more components that support the input / output port(s) 508, such as, but not limited to, a switch, a push button, a pressure pad, a float switch, a keypad, a radio receiver, or a combination thereof.

[0044] The network adapter 512 can be any type of network interface controller configured to provide a means of communication over a network 520 with another computing device, such as a remote computing device 522 (e.g., the fault detection processor 102). For example, the remote computing device 522 can be a user device, such as a mobile phone, smartphone, tablet, laptop, or a combination thereof, or another computing device. The power supply 514 is configured to convert high-voltage alternating current (e.g., AC) to direct current (e.g., DC) to provide power to the other components (e.g., the processor 504, the display adapter 506, the one or more input / output port(s) 508, the one or more input / output component(s) 510, the network adapter 512, and the memory 516) of the computing device 502.

[0045] Additionally, Memory 516 can be a mass storage device and / or system memory, such as a hard disk drive, a memory card, a solid-state drive, random access memory (RAM), or a combination thereof. Memory 516 is configured to provide storage for instructions and data associated with the operation of the computing device 502. Memory 516 can generally include an operating system 524, recognition software 526, and recognition data 528. For example, the operating system 524 is configured to manage and / or process any of the data and / or instructions associated with the recognition software 526 and / or the recognition data 528, as described in more detail in this document.

[0046] Furthermore, a system bus 530 is included within the computing device 502, which is configured to couple each of the various components (e.g., the processor 504, the display adapter 506, the one or more input / output ports 508, the one or more input / output component(s) 510, the network adapter 512, the power supply 514, and the memory 516) of the computing device 502. It is also understood that each of the components of the computing device 502 and the functionality associated with each component of the computing device 502 can be implemented within the remote computing device 522. While the operating environment, which is in Fig. Figure 5 illustrates a specific configuration that is associated with at least the computing device 502, the network 520 and the remote computing device 522; it is understood that the operating environment can be configured in any way.

[0047] Thus, one or more examples of the present disclosure provide a means for inspecting a vehicle based on a real-time video stream of one or more vehicle components, captured by a mixed-media device worn by an operator. One or more machine learning models are configured to detect, based on the real-time video stream, whether one or more vehicle components are associated with a defect. The detection of one or more defects is communicated to the operator via the mixed-media device, which also guides the operator on how to address the one or more defects.

[0048] Unless expressly stated otherwise in this document, all numerical values ​​indicating mechanical / thermal properties, percentages of compositions, dimensions and / or tolerances, or other parameters are to be understood as modified by the word "approximately" or "about" when describing the scope of this disclosure. This modification is desirable for various reasons, including industrial practice, material, manufacturing and assembly tolerances, and testability.

[0049] As used herein, the phrase "at least one of A, B and C" should be interpreted as meaning a logical (A OR B OR C) using a non-exclusive logical OR, and should not be interpreted as meaning "at least one of A, at least one of B and at least one of C".

[0050] In this application, the terms "controller" and / or "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinable logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the foregoing, such as in a system-on-a-chip.

[0051] The term storage is a subset of the term computer-readable medium. The term computer-readable medium, as used here, does not include transitory electrical or electromagnetic signals that propagate through a medium (such as via a carrier wave); the term computer-readable medium can therefore be considered tangible and non-transient.Non-restrictive examples of a non-transitory, tangible, computer-readable medium include non-volatile memory circuits (such as a flash memory circuit, a erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0052] The devices and procedures described in this application can be implemented in whole or in part by a specialized computer created by configuring a general-purpose computer to perform one or more specific functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of an experienced technician or programmer.

[0053] The description of the revelation is purely exemplary, and thus it is intended that examples which do not deviate from the content of the revelation fall within its scope. Such variations are not to be considered a deviation from the nature and scope of the revelation.

[0054] According to the present invention, one or more non-transitory computer-readable media store processor-executable instructions which, when executed by at least one processor, cause the at least one processor to: receive, by a processor, a video stream from a mixed-media device that captures one or more components of a vehicle; detect, by an algorithm associated with the processor, one or more defects associated with the one or more components of the vehicle in response to the receipt of the video stream; transmit, via the mixed-media device, dynamic feedback to an operator of the mixed-media device from the processor based on the detection of the one or more defects; and instruct the operator, via the mixed-media device, to address the one or more defects based on the transmission of the dynamic feedback.

[0055] According to one embodiment, the video stream comprises one or more images within a field of view of the mixed-media device.

[0056] According to one embodiment, dynamic feedback includes one or more instructions, and wherein the at least one processor is further caused to: transmit the one or more instructions to the operator via the mixed-media device, wherein the one or more instructions facilitate steering the field of view of the mixed-media device to successfully complete an inspection of the one or more components.

[0057] According to one embodiment, the dynamic feedback is displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more transducers of the mixed-media device, or a combination thereof.

[0058] According to one embodiment, the at least one processor that is caused to transmit the dynamic feedback to the operator is further caused to do the following: cause the operator to be shown a dynamic update of the user interface, wherein the dynamic update indicates a proper installation of the one or more components via one or more bounding boxes associated with the one or more components, a color associated with the proper installation of the one or more components, a color change indicating that the operator has dealt with the one or more defects, or a combination thereof.

Claims

[1] Procedure, encompassing: Receiving, by a processor, a video stream from a mixed-media device that captures one or more components of a vehicle; Detecting, by an algorithm assigned to the processor, one or more defects associated with one or more components of the vehicle, in response to receiving the video stream; Transmitted, via the mixed-media device, as dynamic feedback to an operator of the mixed-media device from the processor based on the detection of one or more defects; and Instructing the operator, via the mixed-media device, to handle one or more defects based on the transmission of dynamic feedback. [2] Method according to claim 1, wherein the video stream comprises one or more images within a field of view of the mixed-media device. [3] The method of claim 2, wherein the dynamic feedback includes one or more instructions, the method further comprising: Transmitting one or more instructions to the operator via the mixed-media device, wherein the one or more instructions facilitate the steering of the mixed-media device's field of view to successfully complete an inspection of one or more components. [4] Method according to claim 1, wherein the dynamic feedback is displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more transducers of the mixed-media device, or a combination thereof. [5] Method according to claim 4, wherein the transmission of the dynamic feedback to the operator further comprises: To cause the operator to be shown a dynamic update of the user interface, wherein the dynamic update indicates a proper installation of one or more components via one or more bounding boxes associated with the one or more components, a color associated with the proper installation of the one or more components, a color change indicating that the operator has dealt with the one or more defects, or a combination thereof. [6] The method of claim 4, wherein the dynamic feedback includes one or more augmented reality images, the method further comprising: Generating one or more augmented reality images based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions specifying the proper installation of the one or more components; and Displaying one or more augmented reality images on the user interface of the mixed media device. [7] Method according to claim 1, wherein the dynamic feedback includes a first set of one or more video instructions or a second set of one or more video instructions, the method further comprising: Transmitted, via the mixed-media device, the first set of one or more video instructions to the operator based on the captured one or more components, wherein the first set of one or more video instructions includes one or more steps for the proper installation of the one or more components; or Transmitted, via the mixed-media device, the second set of one or more video instructions to the operator based on the detection of one or more defects, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and one or more steps for properly installing the one or more components. [8] Method according to claim 1, wherein the dynamic feedback includes one or more voice notes, the method further comprising: Received by the processor, one or more voice notes based on a voice command from the operator, wherein the one or more voice notes are associated with the video stream from the mixed-media device, and wherein the voice command specifies one or more defects detected by an operator. [9] System, comprehensive: a processor configured to do the following: Receiving a video stream from a mixed-media device that captures one or more components of a vehicle, Detecting, by means of an algorithm assigned to the processor, one or more defects, which are assigned to one or more components of the vehicle, in response to receiving the video stream, Transmitted via the mixed-media device, dynamic feedback is sent from the processor to an operator of the mixed-media device based on the detection of one or more defects and Instructing the operator, via the mixed-media device, to handle one or more defects based on the transmission of dynamic feedback; and the mixed-media device configured to: Transmitting the video stream, receiving the dynamic feedback and Communicating dynamic feedback to the operator, with the dynamic feedback being displayed on a user interface of the mixed-media device, is audibly received by the operator via one or more converters of the mixed-media device, or a combination thereof. [10] System according to claim 9, wherein the video stream comprises one or more images within a field of view of the mixed-media device. [11] System according to claim 10, wherein the dynamic feedback includes one or more instructions, the processor is further configured to: Transmitting one or more instructions to the operator via the mixed-media device, wherein the one or more instructions facilitate the steering of the mixed-media device's field of view to successfully complete an inspection of one or more components. [12] System according to claim 9, wherein the processor configured to transmit dynamic feedback to the operator is further configured to: To cause the operator to be shown a dynamic update of the user interface, wherein the dynamic update indicates a proper installation of one or more components via one or more bounding boxes associated with the one or more components, a color associated with the proper installation of the one or more components, a color change indicating that the operator has dealt with the one or more defects, or a combination thereof. [13] System according to claim 9, wherein the dynamic feedback includes one or more augmented reality images, the processor is further configured to: Generating one or more augmented reality images based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions specifying the proper installation of the one or more components; and Displaying one or more augmented reality images on the user interface of the mixed media device. [14] System according to claim 9, wherein the dynamic feedback includes a first set of one or more video instructions or a second set of one or more video instructions, the processor further configured to: Transmitted, via the mixed-media device, the first set of one or more video instructions to the operator based on the captured one or more components, wherein the first set of one or more video instructions includes one or more steps for the proper installation of the one or more components; or Transmitted, via the mixed-media device, the second set of one or more video instructions to the operator based on the detection of one or more defects, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and one or more steps for properly installing the one or more components. [15] System according to claim 9, wherein the dynamic feedback includes one or more voice notes, the processor is further configured to: Received by the processor, one or more voice notes based on a voice command from the operator, wherein the one or more voice notes are associated with the video stream from the mixed-media device, and wherein the voice command specifies one or more defects detected by an operator.

Images