System and method for inspecting a vehicle

Abstract

The present disclosure provides a system and method for inspecting a vehicle. A method includes receiving a video stream from a hybrid media device that captures one or more components of a vehicle; detecting one or more defects associated with one or more components of the vehicle; transmitting dynamic feedback from a processor to an operator of the mixed media device; and directing the operator to handle the one or more defects via the mixed media device.

Description

Technical Field

[0001] This disclosure relates to inspecting one or more components of a vehicle, and more specifically to systems and methods for inspecting a vehicle using a hybrid media device. Background Technology

[0002] The statements in this section are provided only as background information in connection with this disclosure and may not constitute prior art.

[0003] Warranty and in-house repair issues associated with vehicle production typically involve incorrectly installed parts, rather than defective parts. Because verifying that parts are installed correctly can be considered an additional manufacturing-related task, the production process may need to be rebalanced to accommodate this verification. However, without real-time verification of correct part installation, inefficiencies such as waste, rework, unsatisfactory communication, or a combination thereof can occur in the production process.

[0004] This disclosure addresses these and other issues related to vehicle inspection. Summary of the Invention

[0005] This section provides a general overview of this disclosure and is not a full disclosure of its entire scope or all its features.

[0006] This disclosure provides a method comprising: receiving a video stream from a hybrid media device that captures one or more components of a vehicle by a processor; detecting one or more defects associated with the one or more components of the vehicle by means of an algorithm associated with the processor in response to receiving the video stream; transmitting dynamic feedback from the processor to an operator of the hybrid media device via the hybrid media device based on the detection of the one or more defects; and guiding the operator to process the one or more defects based on the transmission of the dynamic feedback via the hybrid media device; wherein the video stream comprises one or more images within the field of view of the hybrid media device; wherein the dynamic feedback comprises one or more instructions, the method further comprising: transmitting the one or more instructions to the operator via the hybrid media device, wherein the one or more instructions help to guide the field of view of the hybrid media device to successfully complete the inspection of the one or more components; wherein the dynamic feedback is displayed on a user interface of the hybrid media device and is audibly received by the operator via one or more transducers of the hybrid media device, or a combination thereof; wherein transmitting the dynamic feedback to the operator further comprises: causing a dynamic update of the user interface to be displayed to the operator, wherein the dynamic update is via one or more transducers associated with the one or more components. A bounding box, a color associated with the correct installation of the one or more components, a color change indicating that the operator has addressed the one or more defects, or a combination thereof, to indicate the correct installation of the one or more components; wherein the dynamic feedback includes one or more augmented reality images, the method further includes: generating the one or more augmented reality images based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions indicating the correct installation of the one or more components; and overlaying the one or more augmented reality images onto the user interface of the hybrid media device; wherein the dynamic feedback includes a first set of one or more video instructions or a second set of one or more video instructions, the method further includes: transmitting the first set of one or more video instructions to the operator via the hybrid media device based on the captured one or more components, wherein the first set of one or more video instructions includes one or more steps for correctly installing the one or more components; or transmitting the second set of one or more video instructions to the operator via the hybrid media device based on the detection of the one or more defects, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and the one or more steps for correctly installing the one or more components;Furthermore, the dynamic feedback includes one or more voice annotations, and the method further includes: receiving the one or more voice annotations by the processor based on a voice command from the operator, wherein the one or more voice annotations are associated with the video stream from the hybrid media device, and wherein the voice command indicates one or more defects detected by the operator.

[0007] This disclosure provides a system including a processor configured to: receive a video stream from a hybrid media device capturing one or more components of a vehicle; in response to receiving the video stream, detect one or more defects associated with the one or more components of the vehicle using an algorithm associated with the processor; based on the detection of the one or more defects, transmit dynamic feedback from the processor to an operator of the hybrid media device via the hybrid media device; and guide the operator via the hybrid media device to handle the one or more defects based on the transmission of the dynamic feedback; and the hybrid media device is configured to: transmit the video stream, receive the dynamic feedback, and communicate the dynamic feedback to the operator, wherein the dynamic feedback is displayed on a user interface of the hybrid media device and audibly received by the operator via one or more transducers of the hybrid media device, or a combination thereof; wherein the video stream includes one or more images within the field of view of the hybrid media device; wherein the dynamic feedback includes one or more instructions, and the processor is further configured to: transmit the one or more instructions to the operator via the hybrid media device, wherein the one or more instructions help to guide the field of view of the hybrid media device to successfully complete the handling of the one or more defects. The processor, configured to transmit the dynamic feedback to the operator, is further configured to: cause a dynamic update to the user interface to be displayed to the operator, wherein the dynamic update indicates the correct installation of the one or more components via one or more bounding boxes associated with the one or more components, colors associated with the correct installation of the one or more components, color changes indicating that the operator has addressed the one or more defects, or a combination thereof; wherein the dynamic feedback includes one or more augmented reality images, and the processor is further configured to: generate the one or more augmented reality images based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions indicating the correct installation of the one or more components; and overlay the one or more augmented reality images onto the user interface of the hybrid media device; wherein the dynamic feedback includes a first set of one or more video instructions or a second set of one or more video instructions, and the processor is further configured to: transmit the first set of one or more video instructions to the operator via the hybrid media device based on the captured one or more components, wherein the first set of one or more video instructions includes one or more steps for the correct installation of the one or more components;Alternatively, based on the detection of the one or more defects, a second set of one or more video instructions is transmitted to the operator via the hybrid media device, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and one or more steps for correctly installing the one or more components; and wherein the dynamic feedback includes one or more voice annotations, and the processor is further configured to receive the one or more voice annotations based on the operator's voice commands, wherein the one or more voice annotations are associated with the video stream from the hybrid media device.

[0008] This disclosure provides one or more non-transitory computer-readable media storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to: receive a video stream from a hybrid media device capturing one or more components of a vehicle; in response to receiving the video stream, detect one or more defects associated with the one or more components of the vehicle by means of an algorithm associated with the processor; based on the detection of the one or more defects, transmit dynamic feedback from the processor to an operator of the hybrid media device via the hybrid media device; and guide the operator via the hybrid media device to handle the one or more defects based on the transmission of the dynamic feedback; wherein the video stream includes one or more images within the field of view of the hybrid media device; wherein the dynamic feedback includes one or more instructions, and wherein the at least one processor It is also caused to transmit the one or more instructions to the operator via the hybrid media device, wherein the one or more instructions help guide the field of view of the hybrid media device to successfully complete the inspection of the one or more components; wherein the dynamic feedback is displayed on the user interface of the hybrid media device and is audibly received by the operator via one or more transducers of the hybrid media device, or a combination thereof; and wherein the at least one processor caused to transmit the dynamic feedback to the operator is also caused to cause a dynamic update of the user interface to be displayed to the operator, wherein the dynamic update indicates the correct installation of the one or more components via one or more bounding boxes associated with the one or more components, colors associated with the correct installation of the one or more components, color changes indicating that the operator has addressed the one or more defects, or a combination thereof.

[0009] Further applicability will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0010] To better understand this disclosure, various forms of the disclosure will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 This is a block diagram of a vision system associated with the inspection of one or more components of a vehicle according to one or more embodiments of this disclosure; Figure 2 This is an exemplary environment related to a vision system according to one or more embodiments of this disclosure; Figure 3 This is a flowchart illustrating an exemplary method for performing an inspection of one or more components associated with a vehicle, according to one or more embodiments of the present disclosure; Figure 4 This is a flowchart illustrating another exemplary method for performing an inspection of one or more components associated with a vehicle, according to one or more embodiments of this disclosure; and Figure 5 This is a block diagram illustrating an exemplary computer system according to one or more embodiments of the present disclosure.

[0011] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way. Detailed Implementation

[0012] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses. It should be understood that throughout the drawings, corresponding reference numerals indicate the same or corresponding parts and features.

[0013] The examples described herein provide systems and methods for vehicle inspection that can reduce the tasks performed by an operator in real time and allow for more optimized processes (e.g., correct angles, sufficient lighting, sufficient proximity to the vehicle, etc.) to be performed. For example, real-time vehicle inspection can be performed during the production process. Integration of real-time video collection into the operator's role (e.g., proximity installation) can also be provided, which may include advantages for providing enhancements for image collection. Additionally, real-time streaming of video and / or images to a computer device via Wi-Fi, 5G Bluetooth®, or other means can be provided for inference of the video and / or images.

[0014] In one or more examples, the source of video and / or images may be one or more cameras on the mixed media device. Additionally, besides the result of the inspection (e.g., success or otherwise), the screen of the mixed media device may provide the operator with feedback related to the inspection that should be completed, as well as indications of when the inspection is complete. In one or more examples, guidance may be provided to the operator, directing their gaze to perform the next task and / or inspection (e.g., audibly and / or visually). Moreover, immediate feedback may be provided to the operator for unsuccessful inspections, allowing the operator to correct errors, use one or more voice commands to flag problems for downstream repairs during production, and / or use one or more voice commands to provide feedback to the system that the inspection was unsuccessful.

[0015] One or more examples provide the capability for override inspections, which allows the system to save source video and route it to an AI training platform for additional model training. One or more photos of potential points of interest can also be sent to another party and paired with comments in the form of feedback. One or more examples may include capturing one or more still photos of unsuccessful inspections and routing them to downstream workstations for repair, as well as the option for operators to add one or more voice annotations to repair technicians to take various actions (e.g., calling a specific person for details associated with the inspection).

[0016] One or more examples may also provide the ability to use voice tools to collect operator feedback; continuously reinforce ideas; provide one or more alerts to operators; capture videos for operator training; collect training videos during the production process; broadcast to operators so they know a vehicle is approaching; provide model blending; provide operators with a list of tasks and / or instructions on when actions in the production process have been performed (e.g., complete, bypass, etc.); provide guidance during events (e.g., natural disasters, active shooters, etc.); or integrate with one or more learning models.

[0017] Figure 1 It shows the relationship with and vehicles (e.g., such as) Figure 2 The exemplary vision system 100 is used for the inspection of one or more components associated with the vehicle 214 shown. However, it should be understood that the vision system 100 can be applied to inspection processes associated with any structure and is not limited to the inspection of the vehicle 214 and / or one or more components associated with the vehicle 214.

[0018] In one or more embodiments, inspection of one or more components of vehicle 214 is performed by a hybrid media device (e.g., such as...) communicating with defect detection processor 102. Figure 2The hybrid media device 200 shown is supported. The defect detection processor 102 is configured to detect one or more defects associated with one or more components of the vehicle 214 in response to receiving a video stream (e.g., a live or live video stream) originating from the hybrid media device 200. As an example, the hybrid media device 200 can be any electronic device wearable by an operator (e.g., a human operator), such as, but not limited to, glasses, a head-mounted camera, or a body camera. It should be understood that the defect detection processor 102 can be located partially or entirely outside the hybrid media device 200, either on a computer (e.g., an edge PC) separate from the hybrid media device 200, or partially or entirely within the hybrid media device 200.

[0019] refer to Figure 2 The hybrid media device 200 may include one or more lenses 202, one or more sensors 204, one or more housings 206, and one or more arms 208. It should be understood that, although... Figure 2 The hybrid media device 200 shown depicts dual lenses 202, dual sensors 204, dual housings 206 (e.g., housings), and dual arms 208; however, the hybrid media device 200 may include any number (e.g., more or less two) of lenses 202, sensors 204, housings 206, and / or arms 208. In one or more embodiments, one or more lenses 202 are configured to display content to an operator of the hybrid media device 200. However, it should be understood that the operator of the hybrid media device 200 can also view any settings located outside the one or more lenses 202 within the operator's field of vision 216 by viewing settings via the one or more lenses 202. In other words, displaying content to the operator of the hybrid media device 200 does not impede the ability to view settings outside the one or more lenses 202 that are already within the operator's field of vision 216 of the hybrid media device 200 (e.g., viewed via lenses 202).

[0020] One or more sensors 204 are disposed on the hybrid media device 200 near one or more lenses 202 and are configured to capture the operator's field of view 216. However, it should be understood that the one or more sensors 204 may be disposed on or integrated into any part of the construction associated with the hybrid media device 200. The one or more sensors 204 may have one or more camera, lidar, radar, and / or ultrasonic capabilities.

[0021] Additionally, one or more housings 206 may include one or more transducers 210 and / or one or more receivers 212. As an example, one or more transducers 210 may be a speaker array configured to output audio sound waves to an operator of the hybrid media device 200. As another example, one or more receivers 212 may be a microphone array configured to receive one or more voice commands (e.g., one or more voice annotations) from an operator. For example, one or more voice commands may include feedback observed by the operator in relation to one or more components, which provides contextual feedback to the defect detection processor 102. As another example, contextual feedback may enhance the processing of input 104 through prediction algorithm 106, as described herein. One or more housings 206 are attached to one or more arms 208 of the hybrid media device 200. However, it should be understood that one or more housings 206 may be attached to any part of the construction associated with the hybrid media device 200.

[0022] Return to reference Figure 1 The video stream and / or one or more voice commands are received by the defect detection processor 102 as input 104. For example, input 104 is wirelessly transmitted from the hybrid media device 200 to the defect detection processor 102. As another example, the transmission of input 104 is carried out by a wireless communication protocol (e.g., Bluetooth). ® The device supports wireless protocols such as cellular protocols, Wi-Fi protocols, Near Field Communication (NFC) protocols, Ultra Wideband (UWB) protocols, and Li-Fi (Light Fidelity) protocols. As an example, the defect detection processor 102 is also configured to receive one or more voice commands as input 104. The defect detection processor 102 includes a prediction algorithm 106 configured to process the acquired video stream and / or voice commands (e.g., input 104).

[0023] As an example, and based on the processing of input 104, prediction algorithm 106 is configured to stitch together each of the multiple individual images if input 104 is received as multiple individual images. As another example, dynamically received input 104 enables training (e.g., supervised or unsupervised) of prediction algorithm 106. As an additional example, the training of prediction algorithm 106 can be enhanced based on contextual feedback received as one or more voice commands. As yet another example, and based on the processing of input 104, prediction algorithm 106 is configured to analyze input 104 to detect the presence of any defects associated with any of the one or more components. As yet another example, and based on the processing of input 104 and / or the detection of any defects associated with any of the one or more components, prediction algorithm 106 can provide optimized feedback to the operator of hybrid media device 200 by exchanging one or more data elements (e.g., associated with input 104) with one or more machine learning models 108. The one or more machine learning models 108 may include a bounding box model 110, an augmented reality model 112, and a orientation model 114. However, it should be understood that one or more machine learning models 108 may include any number of models related to the inspection of vehicle 214.

[0024] In one or more embodiments, prediction algorithm 106 is configured to process input 104 to determine whether one or more components are incorrectly mounted relative to vehicle 214. In response to prediction algorithm 106 determining the presence of one or more components incorrectly mounted relative to vehicle 214, bounding box model 110 is configured to generate bounding boxes (e.g., virtual bounding boxes) around the incorrectly mounted one or more components and display them by hybrid media device 200, as described in more detail herein. However, it should be understood that bounding box model 110 may generate bounding boxes around each of the one or more components, regardless of whether the one or more components are incorrectly mounted. In cases where bounding box model 110 generates bounding boxes around each of the one or more components, the incorrectly mounted one or more components may be highlighted or emphasized by making the bounding boxes around the incorrectly mounted one or more components appear (e.g., displayed to the operator) as highlighted or emphasized boxes, such as being thicker, brighter, and / or a different color than the bounding boxes generated around each of the one or more correctly mounted components. It should be understood that the bounding box model 110 may highlight one or more incorrectly installed parts in any way that enables the operator to distinguish one or more incorrectly installed parts from one or more correctly installed parts.

[0025] In one or more embodiments, and where any of the one or more components has one or more potential fit orientations, the augmented reality model 112 is configured to generate feedback associated with the correct installation (e.g., correct orientation) of the one or more components having said one or more potential fit orientations. Similarly, where any of the one or more components can be placed in one or more modes, the augmented reality model 112 is configured to generate feedback associated with the correct installation of the one or more components having one or more potentially feasible modes. It should be understood that in any case, the augmented reality model 112 may generate feedback associated with the correct installation of the one or more components in response to the processing of input 104 by the prediction algorithm 106 (e.g., based on the operator's field of vision 216 or from one or more voice commands from the operator).

[0026] In one or more embodiments, prediction algorithm 106 is configured to process input 104 to determine whether one or more components are incorrectly installed relative to vehicle 214. In response to prediction algorithm 106 determining the presence of one or more components incorrectly installed relative to vehicle 214, orientation model 114 is configured to display one or more instructions to an operator of the hybrid media device 200. As an example, the one or more instructions may include procedures for removing the incorrectly installed one or more components, and procedures for correctly installing the one or more components. However, it should be understood that orientation model 114 is configured to display one or more instructions to the operator regardless of whether it is determined that one or more components are incorrectly installed relative to vehicle 214. For example, orientation model 114 may be configured to display one or more instructions to the operator in response to prediction algorithm 106's processing of input 104 (e.g., based on the operator's field of vision 216 or one or more voice commands from the operator). It should also be understood that the one or more instructions may be step-by-step instructions, such as video instructions and / or a displayed list of instructions. It should also be understood that the one or more instructions may be any form of program-related instruction associated with the installation of one or more components.

[0027] As another example, one or more instructions may include one or more directional indicators (e.g., directional arrows) that instruct the operator to change the field of view 216 associated with the settings viewed through one or more lenses 202. For example, one or more directional indicators may instruct the operator to view a component different from the component initially viewed among one or more components. As yet another example, one or more directional indicators may be generated if the inspection of one or more components requires multiple angles, images, or components that cannot be viewed within a single field of view 216. In other words, one or more directional indicators may be generated if the inspection of one or more components requires the operator to move (e.g., change position) around vehicle 214 to capture each desired viewpoint of each of the one or more components being inspected. It should be understood that one or more directional indicators may be provided to the operator based on a global reference to any of the one or more components or a local reference associated with the current position of the hybrid media device 200 relative to the one or more components. One or more directional indicators may be any directional indicator and may be of any type and take different forms and configurations.

[0028] As another example, one or more instructions may also include one or more video-based guides that display the position of any of the one or more components and examples of the correct installation of the one or more components. As yet another example, one or more instructions may also include one or more audio-based guides provided to the operator of the hybrid media device 200 via one or more transducers 210. As another example, one or more instructions may be locked onto any of the one or more components and guide the operator to adjust the field of view 216 based on tracking the locked onto the one or more components.

[0029] In one or more embodiments, the defect detection processor 102 is configured to provide feedback as output 116 to the operator of the hybrid media device 200. As an example, the feedback may be visually displayed on one or more lenses 202 of the hybrid media device 200 and / or audibly provided to the operator via one or more transducers 210 of the hybrid media device 200. For example, a bounding box generated around one or more improperly installed components may be displayed to the operator via the hybrid media device 200. As another example, feedback generated by the augmented reality model 112 may be displayed to the operator via the hybrid media device 200. As yet another example, one or more instructions may be displayed and / or audibly provided to the operator via the hybrid media device 200. In each case, it should be understood that output 116 instructs the operator of the hybrid media device 200 on how to handle (e.g., resolve, mitigate, etc.) an instance of one or more defects associated with one or more components. Vehicle 214. In one or more embodiments, the operator of the hybrid media device 200 can receive a priority list of which of one or more defects should be resolved in an order based on one or more considerations, such as the severity of one or more defects, the speed at which the one or more defects should be resolved, or the proximity of the one or more defects to the operator. For example, the priority list can be generated by prediction algorithm 106.

[0030] Figure 3 This is a flowchart illustrating an exemplary method 300 for inspecting one or more components associated with a vehicle (e.g., vehicle 214) within a vision system (e.g., vision system 100). At operation 302, a video stream (e.g., input 104) is received from a hybrid media device (e.g., hybrid media device 200). For example, the video stream is received at a processor (e.g., defect detection processor 102). As another example, the video stream captures one or more components of the vehicle. As yet another example, the video stream comprises one or more images within the field of view (e.g., field of view 216) of the hybrid media device.

[0031] At operation 304, one or more defects associated with one or more components of the vehicle are detected. For example, one or more defects are detected by an algorithm associated with the processor (e.g., prediction algorithm 106). As another example, one or more defects are detected in response to receiving the video stream.

[0032] At operation 306, dynamic feedback (e.g., output 116) is transmitted from the processor to the operator of the hybrid media device. For example, the dynamic feedback is transmitted to the operator of the hybrid media device via the hybrid media device. As another example, the dynamic feedback is transmitted based on the detection of one or more defects. As yet another example, the dynamic feedback is displayed on the user interface of the hybrid media device and is audibly received by the operator via one or more transducers of the hybrid media device (e.g., one or more transducers 210), or a combination thereof. The user interface may be a dynamic display and / or a communication method, whereby the operator can communicate with the defect detection processor 102, for example, via the hybrid media device 200.

[0033] As another example, a user interface is provided to the operator via one or more lenses (e.g., one or more lenses 202) of the hybrid media device. As an additional example, dynamic feedback includes one or more instructions. In one or more embodiments, one or more instructions are transmitted to the operator. For example, one or more instructions are transmitted to the operator via the hybrid media device. As yet another example, one or more instructions help guide the field of view of the hybrid media device to successfully complete the inspection of one or more components.

[0034] As yet another example, dynamic feedback includes one or more augmented reality images. In another embodiment, one or more augmented reality images are generated based on one or more captured components. In another embodiment, one or more augmented reality images are overlaid onto the user interface of a hybrid media device. For example, one or more augmented reality images include one or more instructions indicating the correct installation of one or more components.

[0035] As another example, dynamic feedback includes a first set of one or more video instructions. In one or more other embodiments, the first set of one or more video instructions is transmitted to an operator. For example, the first set of one or more video instructions is transmitted to the operator via a hybrid media device. As another example, the transmission of the first set of one or more video instructions is based on the captured one or more components. As yet another example, the first set of one or more video instructions includes one or more steps for correctly installing one or more components.

[0036] As another example, dynamic feedback includes a second set of one or more video instructions. In one or more other embodiments, the second set of one or more video instructions is transmitted to an operator. For example, the second set of one or more video instructions is transmitted to the operator via a hybrid media device. As another example, the transmission of the second set of one or more video instructions is based on the detection of one or more defects. As yet another example, the second set of one or more video instructions includes one or more steps for removing one or more components from the vehicle and / or one or more steps for properly installing one or more components.

[0037] As yet another example, dynamic feedback includes one or more voice annotations. In yet another embodiment, the one or more voice annotations are received by a processor. For example, the one or more voice annotations are received based on a voice command from an operator. As yet another example, the one or more voice annotations are associated with a video stream from a mixed media device. As yet another example, a voice command indicates one or more defects detected by an operator.

[0038] In one or more embodiments, dynamic updates to the user interface are displayed to the operator. For example, dynamic updates may indicate correct installation of one or more components via one or more bounding boxes associated with one or more components, colors associated with correct installation of one or more components, indications that the operator has addressed the one or more defects, or a combination thereof. At operation 308, the operator is guided to address one or more defects based on the transmission of dynamic feedback.

[0039] Figure 4 This is a flowchart illustrating another exemplary method 400 for inspecting one or more components associated with a vehicle (e.g., vehicle 214) within a vision system (e.g., vision system 100). At operation 402, a video stream (e.g., input 104) is received from a hybrid media device (e.g., hybrid media device 200). For example, the video stream is received at a processor (e.g., defect detection processor 102). As another example, the video stream captures one or more components of the vehicle.

[0040] At operation 404, the processor performs an analysis to determine if one or more defects are associated with one or more components of the vehicle. If no defects are found, the analysis is performed again (e.g., at a different time, after receiving another input 104, etc.). However, if one or more defects are detected that are associated with one or more components of the vehicle, at operation 406, dynamic feedback (e.g., output 116) is transmitted from the processor to the operator of the hybrid media device. For example, the dynamic feedback is transmitted to the operator of the hybrid media device via the hybrid media device.

[0041] For example, the dynamic feedback transmitted at operation 408 (“Output 1”) includes one or more instructions that help guide the field of view of the hybrid media device to successfully complete the inspection of one or more components. As another example, the dynamic feedback transmitted at operation 410 (“Output 2”) includes one or more augmented reality images generated based on the captured one or more components and then overlaid on the user interface of the hybrid media device. For example, the one or more augmented reality images include one or more instructions indicating the correct installation of one or more components.

[0042] As yet another example, the dynamic feedback transmitted at operation 412 (“Output 3”) includes a first set of one or more video instructions, which includes one or more video-based steps for the proper installation of one or more components. As another example, the dynamic feedback transmitted at operation 414 (“Output 4”) includes a second set of one or more video instructions, which includes one or more video-based steps for removing one or more components from the vehicle and / or one or more steps for the proper installation of one or more components. As an additional example, the dynamic feedback transmitted at operation 416 (“Operation 5”) includes one or more voice annotations based on the operator’s voice command reception, indicating one or more defects detected by the operator. At operation 418, the operator is guided to address one or more defects based on the transmission of the dynamic feedback from operations 408 to 416.

[0043] Figure 5 An operating environment facilitating the execution of one or more systems and methods described herein is illustrated. More specifically, the systems and methods described herein may be implemented using computing device 502. For example, computing device 502 may be a personal computer, desktop computer, laptop computer, tablet computer, handheld computer, server, workstation, mainframe, wearable computer, supercomputer, or a combination thereof. However, it should be understood that the foregoing examples of computing device 502 are not exhaustive, and computing device 502 may be any type of processing or computing device. Computing device 502 typically includes a processor 504, a display adapter 506, one or more input / output ports 508, one or more input / output components 510, a network adapter 512, a power supply 514, and memory 516. However, it should be understood that computing device 502 may include any of the listed components, and is not required to include any of them.

[0044] Processor 504 is configured to provide instructions to computing device 502, enabling computing device 502 to perform one or more tasks, including implementing software programs to perform one or more operations as described in more detail herein. It should also be understood that computing device 502 may include any number of processors 504. Display adapter 506 may be a graphics card or video board that provides computing device 502 with the ability to display content on display device 518. For example, display device 518 may be any screen, monitor, and / or light-emitting component associated with any of a personal computer, desktop computer, laptop computer, tablet computer, handheld computer, server, workstation, host, wearable computer, supercomputer, or a combination thereof. In some examples, display device 502 forms part of hybrid media device 200 and is configured to display a user interface. However, it should be understood that the foregoing examples of display device 518 are non-exhaustive, and display device 518 may be any type of device capable of providing visual display.

[0045] Input / output port 508 provides multiple interfaces (e.g., jacks) for one or more cables to connect to computing device 502. It should be understood that any number of input / output ports 508 may be present on computing device 502. For example, input / output port 508 provides computing device 502 with a means to receive signals and / or data from external devices connected to computing device 502 via one or more cables. As another example, input / output port 508 provides computing device 502 with a means to transmit signals and / or data to external devices connected to computing device 502 via one or more cables. Input / output component 510 may include one or more components supporting input / output port 508, such as, but not limited to, switches, buttons, pressure pads, float switches, keyboards, radio receivers, or combinations thereof.

[0046] Network adapter 512 can be any type of network interface controller configured to provide means for communicating with another computing device (such as remote computing device 522 (e.g., defect detection processor 102)) via network 520. For example, remote computing device 522 can be a user device such as a cellular phone, smartphone, tablet computer, laptop computer or a combination thereof, or other computing device. Power supply 514 is configured to convert high-voltage alternating current (e.g., AC) into direct current (e.g., DC) to provide power to other components of computing device 502 (e.g., processor 504, display adapter 506, one or more input / output ports 508, one or more input / output components 510, network adapter 512, and memory 516).

[0047] Additionally, memory 516 may be a mass storage device and / or system memory, such as a hard disk drive, memory card, solid-state drive, random access memory (RAM), or a combination thereof. Memory 516 is configured to provide storage for instructions and data associated with the operation of computing device 502. Memory 516 may typically include operating system 524, detection software 526, and detection data 528. For example, operating system 524 is configured to manage and / or process any of the data and / or instructions associated with detection software 526 and / or detection data 528, as described in more detail herein.

[0048] Furthermore, a system bus 530 is also included within the computing device 502, configured to couple each of the various components of the computing device 502 (e.g., processor 504, display adapter 506, one or more input / output ports 508, one or more input / output components 510, network adapter 512, power supply 514, and memory 516). It should also be understood that the functions associated with each component of the computing device 502 and with each component of the computing device 502 can be implemented within a remote computing device 522. Although Figure 5 The operating environment shown herein depicts a specific configuration associated with at least computing device 502, network 520, and remote computing device 522; however, it should be understood that the operating environment can be configured in any manner.

[0049] Therefore, one or more examples of this disclosure provide a means for inspecting a vehicle based on a real-time video stream of one or more components of the vehicle captured by a hybrid media device worn by an operator. One or more learning models are configured to detect whether one or more components of the vehicle are associated with a defect based on the real-time video stream. The detection of one or more defects is communicated to the operator via the hybrid media device, thereby guiding the operator on how to handle the one or more defects.

[0050] Unless otherwise expressly indicated herein, all numerical values ​​indicating mechanical / thermal properties, percentage of composition, dimensions and / or tolerances or other characteristics should be understood as being modified by the words “about” or “approximately” when describing the scope of this disclosure. Such modification is desired for various reasons, including: industrial practice; material, manufacturing and assembly tolerances; and testing capabilities.

[0051] As used herein, the phrases A, B, and C at least one should be interpreted as representing logic (A or B or C) using the non-exclusive logic "or", and should not be interpreted as representing "at least one of A, at least one of B, and at least one of C".

[0052] In this application, the terms “controller” and / or “module” may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed analog / digital discrete circuit; digital, analog, or mixed analog / digital integrated circuit; composable logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip.

[0053] The term memory is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not cover transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term computer-readable medium can be considered tangible and non-transient. Non-limiting examples of non-transient tangible computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog magnetic tape or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0054] The apparatus and methods described in this application can be implemented, in part or in whole, by a dedicated computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. Function blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of a technician or programmer.

[0055] The description in this disclosure is merely exemplary in nature, and therefore, variations without departing from the spirit and scope of this disclosure are intended to be made within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

[0056] According to the present invention, one or more non-transitory computer-readable media storing processor-executable instructions, which, when executed by at least one processor, cause the at least one processor to: receive a video stream from a hybrid media device that captures one or more components of a vehicle; in response to receiving the video stream, detect one or more defects associated with the one or more components of the vehicle by means of an algorithm associated with the processor; based on the detection of the one or more defects, transmit dynamic feedback from the processor to an operator of the hybrid media device via the hybrid media device; and guide the operator via the hybrid media device to handle the one or more defects based on the transmission of the dynamic feedback.

[0057] According to one embodiment, the video stream includes one or more images within the field of view of the hybrid media device.

[0058] According to one embodiment, the dynamic feedback includes one or more instructions, and wherein the at least one processor is further caused to transmit the one or more instructions to the operator via the hybrid media device, wherein the one or more instructions help to guide the field of view of the hybrid media device to successfully complete the inspection of the one or more components.

[0059] According to one embodiment, the dynamic feedback is displayed on the user interface of the hybrid media device and is audibly received by the operator via one or more transducers of the hybrid media device, or a combination thereof.

[0060] According to one embodiment, the at least one processor that is caused to transmit the dynamic feedback to the operator is also caused to: cause a dynamic update to the user interface to be displayed to the operator, wherein the dynamic update indicates the correct installation of the one or more components via one or more bounding boxes associated with the one or more components, colors associated with the correct installation of the one or more components, color changes indicating that the operator has addressed the one or more defects, or a combination thereof.

Claims

1. A method comprising: The processor receives video streams from a mixed media device that captures one or more components of a vehicle; In response to receiving the video stream, one or more defects associated with one or more components of the vehicle are detected by an algorithm associated with the processor; Based on the detection of the one or more defects, dynamic feedback is transmitted from the processor to the operator of the hybrid media device via the hybrid media device; as well as The operator is guided via the hybrid media device to address one or more defects based on the transmission of the dynamic feedback.

2. The method of claim 1, wherein the video stream comprises one or more images within the field of view of the hybrid media device.

3. The method of claim 2, wherein the dynamic feedback comprises one or more instructions, and the method further comprises: The one or more instructions are transmitted to the operator via the hybrid media device, wherein the one or more instructions help guide the field of view of the hybrid media device to successfully complete the inspection of the one or more components.

4. The method of claim 1, wherein the dynamic feedback is displayed on the user interface of the hybrid media device and is audibly received by the operator via one or more transducers of the hybrid media device, or a combination thereof.

5. The method of claim 4, wherein transmitting the dynamic feedback to the operator further comprises: This causes dynamic updates to the user interface to be displayed to the operator, wherein the dynamic updates indicate the correct installation of the one or more components via one or more bounding boxes associated with the one or more components, colors associated with the correct installation of the one or more components, color changes indicating that the operator has addressed the one or more defects, or a combination thereof.

6. The method of claim 4, wherein the dynamic feedback comprises one or more augmented reality images, and the method further comprises: The one or more augmented reality images are generated based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions indicating the correct installation of the one or more components; as well as The one or more augmented reality images are overlaid onto the user interface of the hybrid media device.

7. The method of claim 1, wherein the dynamic feedback comprises a first group of one or more video instructions or a second group of one or more video instructions, the method further comprising: Based on the captured one or more components, a first set of one or more video instructions is transmitted to the operator via the hybrid media device, wherein the first set of one or more video instructions includes one or more steps for correctly installing the one or more components; or Based on the detection of the one or more defects, a second set of one or more video instructions is transmitted to the operator via the hybrid media device, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and one or more steps for properly installing the one or more components.

8. The method of claim 1, wherein the dynamic feedback includes one or more voice annotations, and the method further includes: The processor receives one or more voice annotations based on the operator's voice commands, wherein the one or more voice annotations are associated with the video stream from the hybrid media device, and wherein the voice commands indicate one or more defects detected by the operator.

9. A system comprising: Processor, the processor being configured to: Receive video streams from a hybrid media device that captures one or more components of a vehicle; In response to receiving the video stream, one or more defects associated with one or more components of the vehicle are detected by an algorithm associated with the processor; Based on the detection of one or more of the defects, dynamic feedback is transmitted from the processor to the operator of the hybrid media device via the hybrid media device; as well as The operator is guided via the hybrid media device to address one or more defects based on the transmission of the dynamic feedback; and The hybrid media device is configured to: Transmit the video stream, Receive the dynamic feedback, and The dynamic feedback is communicated to the operator, wherein the dynamic feedback is displayed on the user interface of the hybrid media device and is audibly received by the operator via one or more transducers of the hybrid media device, or a combination thereof.

10. The system of claim 9, wherein the video stream comprises one or more images within the field of view of the hybrid media device.

11. The system of claim 10, wherein the dynamic feedback comprises one or more instructions, and the processor is further configured to: The one or more instructions are transmitted to the operator via the hybrid media device, wherein the one or more instructions help guide the field of view of the hybrid media device to successfully complete the inspection of the one or more components.

12. The system of claim 9, wherein the processor configured to transmit the dynamic feedback to the operator is further configured to: This causes dynamic updates to the user interface to be displayed to the operator, wherein the dynamic updates indicate the correct installation of the one or more components via one or more bounding boxes associated with the one or more components, colors associated with the correct installation of the one or more components, color changes indicating that the operator has addressed the one or more defects, or a combination thereof.

13. The system of claim 9, wherein the dynamic feedback comprises one or more augmented reality images, and the processor is further configured to: The one or more augmented reality images are generated based on the captured one or more components, wherein the one or more augmented reality images include one or more instructions indicating the correct installation of the one or more components; and The one or more augmented reality images are overlaid onto the user interface of the hybrid media device.

14. The system of claim 9, wherein the dynamic feedback comprises a first group of one or more video instructions or a second group of one or more video instructions, and the processor is further configured to: Based on the captured one or more components, a first set of one or more video instructions is transmitted to the operator via the hybrid media device, wherein the first set of one or more video instructions includes one or more steps for correctly installing the one or more components; or Based on the detection of the one or more defects, a second set of one or more video instructions is transmitted to the operator via the hybrid media device, wherein the second set of one or more video instructions includes one or more steps for removing the one or more components from the vehicle and one or more steps for properly installing the one or more components.

15. The system of claim 9, wherein the dynamic feedback includes one or more voice annotations, and the processor is further configured to: The processor receives one or more voice annotations based on the operator's voice commands, wherein the one or more voice annotations are associated with the video stream from the hybrid media device, and wherein the voice commands indicate one or more defects detected by the operator.

Images