Multi-camera visual positioning method, device and storage medium

Abstract

Embodiments of the present disclosure disclose a multi-camera visual positioning method, device and storage medium, which determines a normal camera in a current state in a multi-camera positioning system, and then determines whether each normal camera is faulty, and if so, marks it as a faulty camera, and then determines the number of normal cameras, and determines and runs a target positioning mode according to the number of normal cameras, and then determines whether each faulty camera is normal, and if so, marks it as a normal camera, and determines the number of normal cameras, and determines and runs a target positioning mode according to the number of normal cameras, to achieve dynamic degradation and adaptive recovery of the multi-camera positioning system. The method can detect faulty cameras in real time, and continue to provide stable and reliable positioning services using the remaining normal cameras, and can detect whether the faulty cameras are recovered in real time, to achieve the purpose of adaptive switching back to a better mode after recovery.

Description

Technical Field

[0001] This disclosure relates to the field of positioning technology, and in particular to a multi-camera visual positioning method, device and storage medium. Background Technology

[0002] Multi-camera systems offer a wider field of view and richer visual features through multiple perspectives, attracting widespread attention in fields such as autonomous driving and high-precision indoor positioning, in order to overcome the shortcomings of monocular cameras, such as limited field of view and susceptibility to occlusion.

[0003] Related technologies typically employ a fixed configuration and static processing approach, such as the traditional multi-camera visual SLAM (Simultaneous Localization and Mapping) system, which configures a fixed number of cameras at startup and requires all cameras to function properly during runtime.

[0004] However, when a camera fails due to hardware malfunction, obstruction, or data transmission problems, the entire system often cannot continue to work. The lack of runtime fault tolerance mechanisms leads to the entire positioning system shutting down or experiencing a severe performance degradation, making it impossible to continue providing stable and reliable positioning services.

[0005] Therefore, improving positioning accuracy is a research topic that requires continuous effort in the field of autonomous driving. Summary of the Invention

[0006] To address the aforementioned technical problems, or at least partially address them, this disclosure provides a multi-camera visual positioning method, device, and storage medium that can detect camera malfunctions and continue to provide stable and reliable positioning services even in the event of a malfunction, thereby improving the fault tolerance and robustness of the entire system.

[0007] In a first aspect, embodiments of this disclosure provide a multi-camera visual positioning method, the method comprising:

[0008] Identify the normal cameras in the multi-camera positioning system that are currently in a normal state.

[0009] For each of the normal cameras, determine whether the normal camera is malfunctioning. If so, mark the normal camera as a malfunctioning camera whose current state is malfunctioning.

[0010] Determine the number of normal cameras, and based on the number of normal cameras, determine the target positioning mode of the multi-camera positioning system, and run the target positioning mode;

[0011] For each of the faulty cameras, determine whether the faulty camera has returned to normal. If so, mark the faulty camera as a normal camera, determine the number of normal cameras, and determine the target positioning mode of the multi-camera positioning system based on the number of normal cameras, and run the target positioning mode.

[0012] Secondly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the multi-camera visual positioning method as described above.

[0013] Thirdly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the multi-camera visual positioning method as described above.

[0014] This disclosure provides a multi-camera visual positioning method. The method identifies currently functioning cameras in a multi-camera positioning system. For each functioning camera, it determines whether it has malfunctioned. If so, it marks it as a faulty camera. The number of functioning cameras is then determined, and a target positioning mode for the multi-camera positioning system is determined based on this number. This target positioning mode is then run. Similarly, for each faulty camera, it determines whether it has recovered. If so, it is marked as a functioning camera, and the number of functioning cameras is determined. The target positioning mode for the multi-camera positioning system is then run, enabling dynamic degradation and adaptive recovery of the multi-camera positioning system. This method can detect faulty cameras in real-time during the operation of the multi-camera positioning system. When a camera fails due to hardware problems, occlusion, or data transmission issues, the remaining functioning cameras can be used to determine the target positioning mode to continue providing stable and reliable positioning services, solving the problem of overall failure in existing technologies. Furthermore, after detecting a camera malfunction, the method can detect whether the faulty camera has recovered in real-time, using the recovered functioning cameras to determine the target positioning mode, achieving adaptive switching back to a better mode after fault recovery without manual intervention. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a flowchart of a multi-camera visual positioning method according to an embodiment of this disclosure;

[0017] Figure 2 This is a control flowchart for cold start of a multi-camera positioning system provided in an embodiment of this disclosure;

[0018] Figure 3 This is a flowchart of a fault detection process for a reference camera provided in an embodiment of this disclosure;

[0019] Figure 4 This is a flowchart of a non-reference camera detection process provided in an embodiment of this disclosure;

[0020] Figure 5 This is a schematic diagram of resource release provided in an embodiment of this disclosure;

[0021] Figure 6 This is a schematic diagram of dynamic degradation provided in an embodiment of this disclosure;

[0022] Figure 7 This is a flowchart of a backend location thread version check provided in an embodiment of this disclosure;

[0023] Figure 8 This is an interactive schematic diagram of a version check provided in an embodiment of this disclosure;

[0024] Figure 9 This is a flowchart illustrating a process for handling all camera failures provided in an embodiment of this disclosure;

[0025] Figure 10 This is a flowchart of a recovery stability verification method provided in an embodiment of this disclosure;

[0026] Figure 11 This is a schematic diagram illustrating a recovery stability verification method provided in an embodiment of this disclosure;

[0027] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] Figure 1 This is a flowchart illustrating a multi-camera visual positioning method according to an embodiment of this disclosure. The method can be executed by a multi-camera visual positioning device, which can be implemented in software and / or hardware and can be configured in an electronic device. Figure 1 As shown, the method may specifically include the following steps:

[0031] S110. Determine that the current status of the camera in the multi-camera positioning system is normal.

[0032] In this embodiment of the disclosure, the multi-camera positioning system can consist of multiple cameras used for visual positioning. To ensure the stable operation of the multi-camera positioning system, it can be forced to enter multi-camera mode only when all configured cameras are functioning normally when the multi-camera positioning system is started.

[0033] In some implementations, the method provided in this disclosure further includes:

[0034] In response to the detection of a cold start in the multi-camera positioning system, determine whether each camera in the multi-camera positioning system is malfunctioning;

[0035] If there are no faulty cameras in the multi-camera positioning system, the system will operate in multi-camera mode. If there are faulty cameras in the system, the system will return to re-evaluate whether each camera in the system is faulty until the number of evaluations reaches a preset threshold or the evaluation time reaches a preset time threshold, and then generate and display an early warning message.

[0036] Specifically, after detecting a cold start of the multi-camera positioning system, the status of each camera in the multi-camera positioning system can be continuously monitored to determine whether each camera is in a normal state.

[0037] For example, by setting a preset number of cycles and the camera's acquisition cycle, the timeout detection threshold of the camera can be calculated. Then, by determining whether the camera is in a normal state, it can be determined whether the acquired image is obtained within the timeout detection threshold.

[0038] In this embodiment of the disclosure, if there is no faulty camera in the multi-camera positioning system, the multi-camera positioning system can be controlled to enter the multi-camera mode, and then the positioning can be achieved by operating in the multi-camera mode.

[0039] Furthermore, if any camera in the multi-camera positioning system malfunctions, the system can re-evaluate the malfunction of each camera until the number of evaluations reaches a preset threshold or the evaluation duration reaches a preset duration threshold. At this point, an early warning can be generated and displayed to alert the user that the multi-camera positioning system is malfunctioning. The preset number of evaluations can be a pre-set maximum number of times to cyclically detect camera malfunctions, and the preset duration threshold can be a pre-set maximum duration for cyclically detecting camera malfunctions.

[0040] For example, Figure 2 This is a control flowchart for cold start of a multi-camera positioning system provided in an embodiment of this disclosure, such as... Figure 2 As shown, after a system cold start, the multi-camera positioning system can be initialized first to load configuration parameters. These parameters may include camera hardware parameters (such as resolution and frame rate), calibration parameters (such as camera intrinsic and extrinsic parameters), time synchronization and triggering parameters (such as synchronization mode and timestamp accuracy), and system interface parameters (such as data protocol). Further, a cold start check can be performed to determine if all cameras are functioning correctly, i.e., if all cameras are healthy. If so, the system can enter multi-camera mode and use all healthy cameras for positioning. If not, it can enter monitoring mode to continuously check the status of each camera until the number of checks reaches a preset threshold or the check duration reaches a preset threshold.

[0041] In the above implementation method, during the cold start of the multi-camera positioning system, the system checks whether each camera in the multi-camera positioning system is faulty. The system then enters the multi-camera mode when all cameras are functioning normally, ensuring that all cameras can work normally during system operation. This guarantees the reliability of positioning and avoids the system becoming unusable due to dynamic degradation.

[0042] In this embodiment of the disclosure, during the operation of the multi-camera positioning system, a normal camera in the multi-camera positioning system can be identified as currently in a normal state. For example, a camera currently in use in the multi-camera positioning system can be identified as a normal camera, which can be understood as an active camera participating in positioning.

[0043] The current state can be understood as the health status maintained for the camera, i.e., whether it is faulty or normal. In addition to maintaining the camera's health status, it can also maintain the number of consecutive image acquisition failures and the last successful timestamp, etc. In this embodiment, the image acquisition thread can maintain the corresponding current state, the number of consecutive image acquisition failures, and the last successful timestamp for each camera.

[0044] S120. For each normal camera, determine whether the normal camera is faulty. If so, mark the normal camera as a faulty camera whose current state is faulty.

[0045] Specifically, for each normal camera, it is possible to detect whether a normal camera has malfunctioned in real time. If a malfunction is detected, a degradation process can be triggered, allowing the remaining normal cameras to continue positioning. If a normal camera is detected to be malfunctioning, it can be marked as a faulty camera currently in a faulty state.

[0046] In this embodiment of the disclosure, considering that existing systems often use fixed long timeout thresholds (such as 120 seconds) when detecting camera faults, resulting in large fault detection delays and failing to meet the requirements of high real-time applications, a dynamic timeout detection mechanism can also be used to detect the status of each camera to ensure the real-time performance and accuracy of the detection.

[0047] In one specific implementation, determining whether a normal camera has malfunctioned includes:

[0048] Based on the acquisition cycle of the normal camera and the preset number of cycles, determine the timeout detection threshold for the normal camera; if no image is acquired from the normal camera within the timeout detection threshold, it is determined that the normal camera has malfunctioned.

[0049] The acquisition period can be the camera's frame interval. For example, a camera with a frame rate of 20fps has a frame interval of 50ms. 50ms can be understood as the time difference between two consecutive image acquisitions. The preset number of periods can be a pre-set relative number of acquisition periods, such as 2 periods.

[0050] Specifically, the acquisition period of a normal camera can be multiplied by a preset number of periods to obtain the timeout detection threshold for the normal camera. This timeout detection threshold can change with the frame rate of different cameras, thereby achieving the purpose of adaptively adjusting the dynamic detection threshold for different cameras. Compared with using a fixed threshold (such as 120s), this timeout detection threshold can reduce the fault detection latency from minutes to milliseconds.

[0051] In this embodiment of the disclosure, the image acquisition thread can determine in real time whether it has acquired the image of the normal camera within the timeout detection time threshold. If it has, it means that the normal camera is still active and has not malfunctioned, and its current state is still normal. If it has not, it means that the normal camera has malfunctioned. Then the image acquisition thread can mark the normal camera as a faulty camera with a current state of malfunction in order to reset the current state of the normal camera and trigger the subsequent degradation process.

[0052] The above implementation dynamically determines the timeout detection threshold by combining the acquisition cycle of the normal camera with a preset number of cycles. Then, it realizes fault detection of the normal camera based on the timeout detection threshold, achieving the purpose of dynamic timeout detection. Compared with using a fixed threshold, it can adopt a dynamic timeout threshold (such as 2 cycles) that is related to the camera's own frame rate, achieving the purpose of adaptive adjustment for different cameras. This enables millisecond-level rapid fault perception, significantly improves the system response speed, and thus ensures the real-time performance of the detection.

[0053] In this embodiment of the disclosure, during the process of detecting whether each normal camera has malfunctioned, a reference camera can be determined among the normal cameras first, and then the reference camera can be checked for malfunction. If no malfunction occurs, other normal cameras can be checked for malfunction, and the timestamps of the reference cameras can be synchronized.

[0054] For example, Figure 3 This is a flowchart illustrating a fault detection process for a reference camera, as provided in an embodiment of this disclosure. Figure 3 As shown, the image acquisition thread can first acquire images from the reference camera via the camera data pipeline. The reference camera can be randomly selected from all normal cameras, or it can be selected according to a pre-set priority. Furthermore, the image acquisition thread can calculate the timeout detection threshold for the reference camera, that is, calculate the timeout detection threshold using the reference camera's acquisition period and a preset number of periods. For example, if the reference camera's frame interval is 50ms and the preset number of periods is 2, then the timeout detection threshold is 100ms.

[0055] Furthermore, the idle time of the reference camera can be checked, and it can be determined whether the idle time is less than the timeout detection threshold. The idle time can be the duration during which no image is acquired from the reference camera. If the idle time is less than the timeout detection threshold, it means that an image was successfully acquired from the reference camera within the timeout detection threshold. The acquisition timestamp can be recorded, and then each non-reference camera in the normal cameras can be processed in a loop.

[0056] If the idle time is greater than or equal to the timeout detection threshold, it means that the image acquired by the reference camera was not successfully obtained within the timeout detection threshold. At this time, it can be further determined whether the multi-camera positioning system is in dynamic adjustment mode. If so, log is recorded and dynamic degradation processing is triggered, and the failure status is returned. The system continues to run using the remaining normal cameras. If not, it is handled in the traditional mode, the camera timeout flag is set, a fatal error is returned, and the system exits.

[0057] Figure 4 This is a flowchart of a non-reference camera detection process provided in an embodiment of this disclosure, such as... Figure 4As shown, assuming the reference camera is functioning correctly, each non-reference camera among the normal cameras can be processed cyclically. Specifically, the idle time of the non-reference camera can be checked first, and it can be determined whether the idle time is less than the timeout detection threshold. This timeout detection threshold is determined based on the acquisition cycle of the non-reference camera and the preset number of cycles.

[0058] If the idle time of a non-reference camera is greater than or equal to the timeout detection threshold, the system can continue to determine whether the multi-camera positioning system is in dynamic adjustment mode. If so, logs are recorded, dynamic degradation processing is triggered, and a failure status is returned. The system continues to run using the remaining normal cameras. If not, the system is processed in the traditional mode, a camera timeout flag is set, a fatal error is returned, and the system exits.

[0059] If the idle time of a non-reference camera is less than the timeout detection threshold, an attempt is made to acquire an image from that camera using a non-blocking method, and the success of image acquisition is determined. If image acquisition fails, the idle time is accumulated (by adding the frame interval to the idle time), and the timeout is checked again in the next loop. If image acquisition is successful, timestamp synchronization is checked, and the timestamp difference between the non-reference camera and the reference camera is calculated. It is then determined whether this timestamp difference is greater than the synchronization threshold (the synchronization threshold can be determined based on the image interval threshold and the frame interval). If the timestamp difference is greater than the synchronization threshold, it indicates that the non-reference camera is out of sync with the reference camera, and the loop can exit, waiting for the next cycle to continue checking each non-reference camera among the normal cameras.

[0060] If the timestamp difference is less than or equal to the synchronization threshold, it indicates that the timestamps of the non-reference camera and the reference camera are synchronized. The idle time of the non-reference camera can be reset (idle time = 0), and it can be marked as a normal camera. Then, it is determined whether there are other non-reference cameras among the normal cameras. If so, processing of other non-reference cameras among the normal cameras continues; otherwise, it indicates that all non-reference cameras have been successfully synchronized, and a synchronization success status is returned. Through the above process, timestamp synchronization verification between all non-reference cameras and the reference camera can be achieved, ensuring that each camera is aligned to the same timeline as much as possible, guaranteeing that each camera can observe the scene at the same moment, avoiding feature position shifts due to time differences, and thus improving positioning accuracy.

[0061] The various processes involved in the embodiments of this disclosure can be implemented using a multi-threaded architecture. Specifically, the multi-threaded architecture may include an image acquisition thread, an image processing thread group (including image processing threads corresponding to each camera), a backend positioning thread, and an asynchronous worker thread group (including various asynchronous worker threads that execute in parallel).

[0062] The image acquisition thread is used to cyclically acquire images from each camera in the camera pipeline, and perform dynamic timeout detection, that is, to determine whether each normal camera is malfunctioning. If so, the normal camera is marked as a malfunctioning camera. The image acquisition thread can also update the list of normal cameras in real time by managing the status of each camera.

[0063] The image processing thread can be understood as the image processing thread allocated to the camera, which can perform processing steps such as feature extraction and frame construction. The backend localization thread can call the core localization algorithm to perform multi-camera fusion or monocular localization. The asynchronous worker thread group can execute computational tasks such as keyframe processing and local map optimization in parallel.

[0064] In traditional multi-camera vision systems, data processing for each camera typically employs serial or simple parallel pipelines, resulting in computational efficiency bottlenecks. When some cameras fail, the entire system often ceases operation, leading to a waste of computational resources. Therefore, in this embodiment, upon detecting a malfunction in a normal camera, the list of normal cameras can be updated. This list releases the resources occupied by the image processing threads corresponding to the failed cameras, dynamically releasing computational resources (such as threads and CPU cores) and improving overall system performance.

[0065] In one specific implementation, after marking a normal camera as a faulty camera whose current state is faulty, the method further includes:

[0066] Update the normal camera list and broadcast the normal camera list to the image processing thread corresponding to each camera, so that the image processing thread that does not have the information of the corresponding camera stored in the normal camera list and is not in sleep mode will enter sleep mode; the normal camera list stores the information of the normal cameras.

[0067] After marking the faulty camera as a normal camera, the following is also included:

[0068] Update the normal camera list and broadcast the normal camera list to the image processing thread corresponding to each camera, so that the image processing thread that has the corresponding camera information stored in the normal camera list and is in a dormant state will end its dormancy.

[0069] Specifically, after marking normal cameras as faulty cameras, the image acquisition thread can update the normal camera list based on the remaining normal cameras to record all normal cameras. Then, the normal camera list is broadcast to the image processing thread corresponding to each camera. After receiving the normal camera list, the non-sleeping image processing thread determines whether the corresponding camera is in the normal camera list. If not, it automatically enters sleep mode to release the CPU resources occupied by the image processing thread.

[0070] Correspondingly, after marking a faulty camera as a normal camera, the image acquisition thread can update the normal camera list based on the remaining normal cameras, and then broadcast the normal camera list to the image processing thread corresponding to each camera. After receiving the normal camera list, the dormant image processing thread can determine whether the corresponding camera is in the normal camera list. If so, it can end its dormancy to achieve the purpose of automatic thread wake-up.

[0071] For example, Figure 5 This is a schematic diagram of resource release provided in an embodiment of this disclosure, such as... Figure 5 As shown, the image acquisition thread can continuously acquire images from the camera data pipeline. After detecting a malfunction in a normal camera, it can update the current state of the normal camera and update the normal camera list. At the same time, the current version number of the normal camera list is incremented, and then the condition variable is broadcast to notify all threads that the version of the normal camera list has been updated.

[0072] Specifically, in a normal camera, the image processing thread operates in a producer-consumer mode. When the camera is working normally, the image processing thread can perform feature extraction and frame construction. However, if dynamic degradation occurs, such as... Figure 5 If the normal camera corresponding to Image Processing Thread 1 and Image Processing Thread 2 malfunctions, Image Processing Thread 1 and Image Processing Thread 2 will automatically go to sleep, releasing the corresponding CPU resources.

[0073] The above implementation can broadcast the remaining normal cameras to the image processing threads of each camera through the normal camera list, thereby causing the image processing threads corresponding to the faulty cameras to automatically go to sleep, achieving the purpose of releasing the corresponding CPU resources, so that the released resources can be reallocated to other tasks, improving the system's resource utilization.

[0074] The multi-threaded architecture provided in this disclosure not only improves performance, but more importantly, it enables thread-level dynamic resource scheduling and fault isolation, thereby simultaneously improving system resource utilization and fault tolerance.

[0075] S130. Determine the number of normal cameras, and based on the number of normal cameras, determine the target positioning mode of the multi-camera positioning system, and run the target positioning mode.

[0076] Specifically, after detecting faults in each normal camera, the number of remaining normal cameras can be re-determined, and the target positioning mode of the multi-camera positioning system can be determined based on the number of normal cameras, and then the target positioning mode can be run. The target positioning mode can be a multi-camera mode or a single-camera mode.

[0077] For example, after updating the normal camera list, the image acquisition thread can broadcast the normal camera list as a condition variable to the image processing thread group and the backend positioning thread. The backend positioning thread can determine the number of normal cameras based on the normal camera list, and then determine the target positioning mode of the multi-camera positioning system based on the number of normal cameras.

[0078] For example, Figure 6 This is a schematic diagram of dynamic degradation provided in an embodiment of this disclosure, such as... Figure 6 As shown, the image acquisition thread continuously monitors the camera status, detects the image acquisition status of each camera in real time, and performs dynamic timeout detection. That is, it determines whether a camera timeout is detected by a preset number of cycles. If so, it immediately updates the list of normal cameras, removes the faulty camera, increments the corresponding version number, broadcasts a condition variable to notify all threads, and further initiates dynamic degradation processing.

[0079] Specifically, the backend positioning thread can determine the degradation strategy based on the number of remaining normal cameras. If there are two or more remaining normal cameras, the multi-camera mode continues, using the remaining normal cameras. Furthermore, the image acquisition thread can dynamically schedule resources, and the image processing thread corresponding to the failed camera automatically goes to sleep, freeing up CPU resources for active tasks. If there is only one remaining normal camera, the system smoothly degrades to single-camera mode, leveraging the monocular feature of the multi-camera locator to continue providing uninterrupted positioning services. Again, the image acquisition thread can dynamically schedule resources, and the image processing thread corresponding to the failed camera automatically goes to sleep, freeing up CPU resources for active tasks. If there are zero remaining normal cameras, all cameras fail, and the system enters monitoring mode. The image acquisition thread initializes the failure start time for all cameras and records the start time of the entire failure.

[0080] In this embodiment of the disclosure, when a camera malfunctions, the system can automatically and smoothly downgrade from multi-camera mode to single-camera mode (or multi-camera mode with fewer cameras) to continue providing location services, rather than exiting directly, thus achieving seamless connection of location services.

[0081] In this embodiment of the disclosure, considering the possible communication delay between the image acquisition thread and the backend positioning thread, in order to avoid the backend positioning thread using an outdated camera list and waiting for a long time, a version number mechanism can be introduced. The version number is incremented each time the normal camera list is updated, and all threads are notified through a condition variable. When the backend positioning thread is waiting for data, it can check whether the version number of the camera list it records is consistent with the current version number broadcast by the image acquisition thread. If they are inconsistent, it immediately stops waiting for data and data processing, and obtains the latest normal camera list to avoid unnecessary blocking.

[0082] In some implementations, after marking a normal camera as a faulty camera whose current state is faulty, the method further includes:

[0083] Update the current version number of the normal camera list and send the current version number to the backend positioning thread;

[0084] Determine the number of normal cameras, including:

[0085] The backend positioning thread checks whether the version number of the camera list recorded by the backend positioning thread is consistent with the current version number. If they are inconsistent, the backend positioning thread interrupts the positioning process based on the recorded camera list, updates the recorded camera list and the recorded version number based on the normal camera list, and returns to check whether the recorded version number is consistent with the current version number. If they are consistent, the number of normal cameras is determined based on the recorded camera list.

[0086] Specifically, after updating the list of normal cameras and their corresponding current version numbers, the image acquisition thread can send the current version number to the backend positioning thread. Further, upon receiving the current version number, the backend positioning thread compares it with the version numbers in its recorded camera list. If they do not match, the backend positioning thread interrupts its positioning processing based on the recorded camera list; that is, it stops waiting for and processing data from the cameras in the recorded list, updates the recorded camera list and its version number based on the list of normal cameras, and then returns to re-evaluate whether the recorded version number matches the current version number. If they match, the number of normal cameras is determined based on the recorded camera list.

[0087] For example, Figure 7 This is a flowchart of a version check process for a backend location thread provided in an embodiment of this disclosure, such as... Figure 7 As shown, the backend positioning thread can obtain the list of normal cameras broadcast by the image acquisition thread and wait for the condition variable to obtain the latest version number; further, the backend positioning thread can check whether the version number has changed. If it has, it will interrupt the current waiting or processing and update the recorded camera list and version number. If not, it will dynamically switch the positioning mode according to the number of normal cameras.

[0088] Specifically, the backend positioning thread can determine whether there is one or more normal cameras. If there are multiple cameras, it enters multi-camera mode and uses a multi-camera fusion algorithm. If there is only one camera, it enters single-camera mode and uses the multi-camera locator to support monocular features, thereby performing visual positioning calculations. During the calculation process, the backend positioning thread can continuously check the version number.

[0089] In the process of using multi-camera fusion algorithms, multiple asynchronous worker threads in an asynchronous worker thread group can execute in parallel, such as... Figure 7Asynchronous worker threads 1-3 can perform keyframe processing, local map optimization, and other computational tasks, respectively. It should be noted that the asynchronous worker thread group only processes data from normal cameras; data from faulty cameras is not processed.

[0090] Figure 8 This is an interactive schematic diagram of a version check provided in an embodiment of this disclosure, such as... Figure 8 As shown, after the image acquisition thread detects a camera malfunction or recovery, it can update the list of normal cameras and increment the version number, then broadcast the condition variable and update the condition variable using the list of normal cameras. Further, the backend positioning thread reads the current version number and calls the 3D positioning function, passing the recorded version number to the positioning algorithm layer (i.e., the 3D positioning function). The positioning algorithm layer waits for data based on the recorded version number passed by the backend positioning thread, i.e., waits for the data to be ready. During this process, the positioning algorithm layer can check if the version number has changed. If it has, it sets the version number change flag to true, interrupts the wait, and returns the version number change flag. The backend positioning thread then detects the version number change flag, updates the recorded camera list, calls the 3D positioning function again, uses the latest list, waits for the data to be ready, performs positioning, and returns the positioning result. The backend positioning thread processes the positioning result and outputs the pose. If the version number has not changed, the positioning algorithm layer can check if the data is ready, and continue waiting if the data is not ready. Once the data is ready, it exits the wait, performs positioning, and returns the positioning result.

[0091] The above implementation considers that the backend positioning thread may wait for a long time due to the use of an outdated camera list during dynamic reduction and dynamic recovery. By incrementing the version number and checking the version number, the latest normal camera list is obtained, avoiding unnecessary blocking and ensuring the real-time performance and accuracy of positioning. By introducing a normal camera list version number and adding a version check callback to the waiting logic, the problem of the backend positioning thread being blocked due to waiting for outdated data during mode switching is effectively solved, ensuring the system's real-time responsiveness and state consistency. Furthermore, by maintaining a global, dynamically updated normal camera list, all core threads (image acquisition thread, image processing thread, and backend positioning thread) dynamically adjust their working logic based on this list, achieving system-level, thread-safe dynamic resource scheduling.

[0092] S140. For each faulty camera, determine whether the faulty camera has returned to normal. If so, mark the faulty camera as a normal camera, determine the number of normal cameras, and determine the target positioning mode of the multi-camera positioning system based on the number of normal cameras, and run the target positioning mode.

[0093] In this embodiment of the disclosure, after dynamic downgrading and entering target positioning mode, recovery detection can continue to be performed on the faulty camera to achieve dynamic recovery. Specifically, for each faulty camera, it can be determined whether the faulty camera has returned to normal by acquiring the images it has captured.

[0094] Furthermore, if it is determined that the faulty camera has returned to normal, the faulty camera can be marked as a normal camera, and then the number of normal cameras can be re-determined. Based on the number of normal cameras, the target positioning mode can be determined and then the target positioning mode can be run.

[0095] In one specific implementation, determining whether a faulty camera has returned to normal includes the following steps:

[0096] Step 21: Determine whether the images captured by the faulty camera have been reacquired. If so, determine the corresponding stabilization time threshold based on the number of normal cameras.

[0097] Step 22: Perform a recovery stability verification on the faulty camera. If the faulty camera passes the verification within the stable time threshold, it is determined that the faulty camera has recovered to normal.

[0098] Specifically, in step 21, it can be determined whether the images acquired by the faulty camera have been reacquired. If so, the recovery type can be determined based on the number of normal cameras, and then the corresponding stabilization time threshold can be determined based on the recovery type. The stabilization time threshold can be a pre-set time threshold for verifying recovery stability; different recovery types correspond to different stabilization time thresholds. If there is only one normal camera, the recovery type is from monocular to multi-camera recovery; if there are more than one normal camera, the recovery type is incremental multi-camera recovery.

[0099] For example, the stabilization time threshold is 5 seconds for recovery type from monocular to multi-camera, and 3 seconds for recovery type incremental multi-camera (i.e., multi-camera to multi-camera). It should be noted that the purpose of setting different stabilization time thresholds for different recovery types is as follows: considering that recovery from monocular to multi-camera requires switching the positioning mode, while recovery from multi-camera does not require changing the positioning mode, only adjusting the camera used, a longer stabilization time threshold can be set for recovery from monocular to multi-camera to allow more verification time and ensure the reliability of verification for recovery from monocular to multi-camera.

[0100] After obtaining the stabilization time threshold, in step 22, the recovery stability of the faulty camera can be verified. If the faulty camera passes the verification within the stabilization time threshold, it can be determined that the faulty camera has recovered to normal. The recovery stability verification can employ a dual verification mechanism to ensure the camera continues to function normally.

[0101] In the method of this disclosure embodiment, after acquiring the image captured by the faulty camera, the system does not immediately switch modes. Instead, it goes through a stability verification period to confirm that the faulty camera can continue to work normally. Then, it automatically and safely switches back to the better positioning mode without manual intervention, achieving the purpose of smooth degradation and automatic and safe recovery, forming a complete fault-tolerant closed loop.

[0102] For example, Figure 9 This is a flowchart illustrating a process for handling all camera failures provided in an embodiment of this disclosure, such as... Figure 9 As shown, the image acquisition thread can wait for the cameras to recover and continuously check the camera status. First, the image acquisition thread can determine whether the failure duration of all faulty cameras exceeds a preset timeout threshold (e.g., 30 seconds). If so, a fatal error log is output, indicating that all cameras have been inactive for an extended period. This triggers a fatal error, system exit, or a system-level alarm mechanism. The failure duration can be the time during which no images from the faulty cameras have been acquired.

[0103] If the failure duration of all faulty cameras is less than or equal to the preset timeout threshold, then it is further determined whether the images captured by the faulty cameras have been acquired. If they have been acquired, the failure start time of all cameras can be reset, and the recovery check process can be executed to trigger adaptive recovery and enter the recovery stability verification. If they have not been acquired, the list of normal cameras is updated, and the process returns to wait for camera recovery again, continuously checking the camera status.

[0104] Specifically, the stability verification process can be to determine whether several frames of images have been successfully acquired consecutively and run stably for a period of time. If this condition is not met, the system continues to wait for verification to ensure camera stability. If the condition is met, mode recovery is triggered, and initialization is performed according to the recovery type. For example, if the recovery type is from monocular to multi-camera, the state is reset; if the recovery type is incremental multi-camera, incremental initialization is performed. Furthermore, once mode recovery is complete, the system switches back to the better positioning mode, and the image processing thread corresponding to the camera that has returned to normal is awakened.

[0105] It should be noted that, in addition to the above Figure 9 In addition to all the camera failure scenarios shown, there are also cases where some cameras fail, meaning the number of normal cameras is greater than zero. For this situation, continuous monitoring of camera health is also necessary. Specifically, when the number of normal cameras is greater than zero, it can be checked whether images from the faulty cameras have been acquired. If so, a recovery check process is executed, triggering adaptive recovery to verify the recovery stability of the faulty cameras.

[0106] Regarding step 22 above, in some embodiments, the recovery stability verification of the faulty camera includes the following steps:

[0107] Step 221: Based on the acquired images from the faulty camera, determine whether the faulty camera meets the preset stable image acquisition conditions.

[0108] Step 222: If satisfied, update the stability check count of the faulty camera; if not satisfied, reset the stability check count to zero.

[0109] Step 223: Determine whether the stability check count has reached the preset stability check threshold. If yes, the recovery stability verification of the faulty camera is confirmed to be passed. If no, return to the step of determining whether the faulty camera meets the preset stable image acquisition conditions based on the acquired images of the faulty camera.

[0110] In step 221, the recovery stability verification of the faulty camera is performed. First, it can be determined whether the faulty camera meets the preset stable image acquisition conditions based on the acquired images of the faulty camera. The stable image acquisition conditions can be used to determine whether the acquired images of the faulty camera can be stably acquired within a certain period of time.

[0111] Regarding step 221 above, in one example, based on the acquired images from the faulty camera, determining whether the faulty camera meets the preset stable image acquisition conditions includes the following steps:

[0112] Step 2211: Based on the acquired images from the faulty camera, determine the number of consecutive successful frames acquired in this acquisition of images from the faulty camera.

[0113] Step 2212: If the number of consecutive successful frames exceeds the preset stable frame number threshold, and no image acquisition failure of the faulty camera is detected within the preset time window, then the faulty camera is determined to meet the stable image acquisition condition.

[0114] The preset stable frame number threshold can be the number of consecutive frames of successfully acquired images corresponding to the stable image acquisition condition, and the preset time window can be the time window during which no image acquisition failure is detected corresponding to the stable image acquisition condition.

[0115] By determining whether the number of consecutive successful frames exceeds a preset stable frame threshold, and whether no image acquisition failure of the faulty camera is detected within a preset time window, it is possible to determine that the faulty camera meets the conditions for stable image acquisition when no image acquisition failure is detected for a long time and the number of consecutive successful image acquisition frames is large. This achieves the purpose of accurately judging the recovery stability of the faulty camera, avoiding the erroneous recovery of a faulty camera that has temporarily performed normally due to hardware failure or network transmission jitter, reducing the risk of chain failures caused by misjudgment, and thus ensuring the reliability of the subsequent positioning of the entire system.

[0116] Furthermore, in step 222, if the faulty camera meets the preset stable image acquisition conditions, the stability check count of the faulty camera can be updated. The stability check count can be the number of times the stable image acquisition conditions are continuously determined to be met during the stability recovery verification process. If the faulty camera does not meet the stable image acquisition conditions, the stability check count can be reset to zero.

[0117] Furthermore, in step 223, it can be determined whether the stability check count has reached a preset stability check threshold. This preset stability check threshold can be a pre-set stability check count threshold used to determine the stable recovery of the faulty camera. If the stability check count reaches the preset stability check threshold, it is determined that the recovery stability verification of the faulty camera has passed, and the faulty camera has recovered to normal, thereby updating the list of normal cameras; if it has not reached the threshold, the process returns to the step of determining whether the faulty camera meets the preset stable image acquisition conditions based on the acquired images of the faulty camera.

[0118] It should be noted that by determining the stability check count and comparing it with a preset stability check threshold, it is possible to restore a faulty camera to a normal camera if it repeatedly meets the conditions for stable image acquisition, thus ensuring the reliability of fault recovery. Compared to directly restoring a faulty camera to a normal camera when it meets the conditions for stable image acquisition, combining stability check count verification with stability recovery avoids marking faulty cameras that only temporarily perform normally as normal cameras. This ensures the stability of the normal camera list, prevents frequent switching of positioning modes due to repeated camera failures, and ultimately ensures stable system operation.

[0119] For example, Figure 10 This is a flowchart of a recovery stability verification method provided in an embodiment of this disclosure, such as... Figure 10 As shown, firstly, a stabilization time threshold can be set according to the recovery type, and it is determined whether the verification duration is greater than or equal to the stabilization time threshold. The verification duration can be the duration of the recovery stability verification. If the verification duration is greater than or equal to the stabilization time threshold, it is determined whether the stability check count is greater than or equal to the preset stability check threshold. If yes, mode recovery is triggered, and initialization operations are performed according to the recovery type. If no, recovery is canceled, and the faulty camera still has a fault.

[0120] If the verification duration is less than the stabilization time threshold, the system checks the number of consecutive successful frames and whether image acquisition failed within the preset time window to determine if the stable image acquisition condition is met. If met, the stability check count is incremented; otherwise, it is reset to 0. Further, the system checks if the stability check count is greater than or equal to the preset stability check threshold. If yes, the normal camera list is updated, mode recovery is triggered, and initialization operations are performed based on the recovery type. If no, the system returns to re-evaluate whether the verification duration is greater than or equal to the stabilization time threshold.

[0121] The above implementation method updates the stability check count when the faulty camera meets the conditions for stable image acquisition, and then determines whether the faulty camera has passed the recovery stability verification by using the stability check count. This enables a dual verification mechanism based on stable image acquisition conditions and stability check count, ensuring the reliability of the verification, avoiding erroneous recovery of the faulty camera, and thus further guaranteeing the reliability of positioning.

[0122] In this embodiment of the disclosure, considering that the number of normal cameras may change during the recovery stability verification process of the faulty camera, for example, the number of normal cameras may increase or decrease, the recovery can be canceled when the number of normal cameras decreases (a normal camera is detected as faulty) in order to avoid errors in the stability time threshold during the recovery stability verification process, thereby avoiding erroneous verification.

[0123] For example, Figure 11 This is a schematic diagram of a recovery stability verification provided in an embodiment of this disclosure, such as... Figure 11 As shown, after detecting the image acquired by the faulty camera, the system first checks whether the automatic recovery function is enabled. If not, recovery is not performed. If enabled, it checks whether the number of previously normal cameras has been initialized. If not, it initializes the number of previously normal cameras. If initialized, the recovery type is determined based on the number of normal cameras. Further, it checks whether the number of normal cameras (i.e., the number of normal cameras) changes during recovery. If not, it continues to the recovery stability verification. If it changes, it further checks whether the number of normal cameras increases during recovery. If the number increases, it continues to the recovery stability verification. If the number decreases, it checks whether recovery is in progress. If recovery is in progress, recovery is canceled, all recovery states are reset, the recovery type is set to none, and the stability check count is set to 0. If recovery has not started, the number of previously normal cameras is updated to restart the entire process.

[0124] The multi-camera visual positioning method provided in this embodiment identifies the currently functioning cameras in the multi-camera positioning system. For each functioning camera, it determines whether it has malfunctioned. If so, it marks it as a faulty camera. The number of functioning cameras is then determined, and the target positioning mode of the multi-camera positioning system is determined based on this number. This target positioning mode is then run. Similarly, for each faulty camera, it determines whether it has recovered. If so, it is marked as a functioning camera, and the number of functioning cameras is determined. The target positioning mode of the multi-camera positioning system is then run, enabling dynamic degradation and adaptive recovery of the multi-camera positioning system. This method can detect faulty cameras in real time during the operation of the multi-camera positioning system. When a camera fails due to hardware problems, occlusion, or data transmission issues, the remaining functioning cameras can be used to determine the target positioning mode to continue providing stable and reliable positioning services, solving the problem of overall failure in existing technologies. Furthermore, after detecting a camera malfunction, this method can detect whether the faulty camera has recovered in real time, using the recovered functioning cameras to determine the target positioning mode, achieving adaptive switching back to a better mode after fault recovery without manual intervention.

[0125] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 12 It shows a schematic diagram of a structure suitable for implementing the electronic device 500 in the embodiments of this disclosure. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0126] like Figure 12 As shown, the electronic device 500 may include a processing device 501, a ROM 502, a RAM 503, a bus 504, an input / output (I / O) interface 505, an input device 506, an output device 507, a storage device 508, and a communication device 509. The processing device (e.g., a central processing unit, a graphics processor, etc.) 501 can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in the read-only memory (ROM) 502 or a program loaded from the storage device 508 into the random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing device 501, the ROM 502, and the RAM 503 are interconnected via the bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0127] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the multi-camera visual positioning method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0128] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0129] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform any of the methods provided in the embodiments of this disclosure.

[0130] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A multi-camera visual positioning method, characterized in that, The method includes: Identify the normal cameras in the multi-camera positioning system that are currently in a normal state. For each of the normal cameras, determine whether the normal camera is malfunctioning. If so, mark the normal camera as a malfunctioning camera whose current state is malfunctioning. Determine the number of normal cameras, and based on the number of normal cameras, determine the target positioning mode of the multi-camera positioning system, and run the target positioning mode; For each of the faulty cameras, determine whether the faulty camera has returned to normal. If so, mark the faulty camera as a normal camera, determine the number of normal cameras, and determine the target positioning mode of the multi-camera positioning system based on the number of normal cameras, and run the target positioning mode.

2. The method according to claim 1, characterized in that, Determining whether the normally functioning camera is malfunctioning includes: The timeout detection threshold of the normal camera is determined based on the acquisition cycle of the normal camera and the preset number of cycles. If no image is acquired from the normal camera within the timeout detection threshold, the normal camera is determined to be faulty.

3. The method according to claim 1, characterized in that, The process of determining whether the faulty camera has returned to normal includes: Determine whether the image captured by the faulty camera has been reacquired. If so, determine the corresponding stabilization time threshold based on the number of normal cameras. The faulty camera is subjected to a recovery stability verification. If the faulty camera passes the verification within the duration of the stability time threshold, it is determined that the faulty camera has returned to normal.

4. The method according to claim 3, characterized in that, The process of restoring stability verification for the faulty camera includes: Based on the acquired images from the faulty camera, determine whether the faulty camera meets the preset stable image acquisition conditions; If the condition is met, the stability check count of the faulty camera is updated; if the condition is not met, the stability check count is reset to zero. Determine whether the stability check count has reached the preset stability check threshold. If yes, then the recovery stability verification of the faulty camera has passed. If no, return to the step of determining whether the faulty camera meets the preset stable image acquisition conditions based on the acquired images of the faulty camera.

5. The method according to claim 4, characterized in that, Based on the acquired images from the faulty camera, determine whether the faulty camera meets preset stable image acquisition conditions, including: Based on the acquired images from the faulty camera, determine the number of consecutive successful frames acquired in this acquisition of the faulty camera's images; If the number of consecutive successful frames exceeds a preset stable frame count threshold, and no image acquisition failure of the faulty camera is detected within a preset time window, then the faulty camera is determined to meet the stable image acquisition condition.

6. The method according to claim 1, characterized in that, After marking the normal camera as a faulty camera whose current state is faulty, the method further includes: Update the normal camera list and broadcast the normal camera list to the image processing thread corresponding to each camera, so that the image processing thread that does not have the information of the corresponding camera stored in the normal camera list and is not in a sleep state will enter a sleep state; the normal camera list stores the information of the normal cameras. After marking the faulty camera as the normal camera, the method further includes: The normal camera list is updated and broadcast to the image processing thread corresponding to each camera, so that the image processing thread whose information of the corresponding camera is stored in the normal camera list and is in a dormant state ends its dormancy.

7. The method according to claim 6, characterized in that, After marking the normal camera as a faulty camera whose current state is faulty, the method further includes: Update the current version number of the normal camera list and send the current version number to the backend positioning thread; Determining the number of normal cameras includes: The backend positioning thread determines whether the version number of the camera list recorded by the backend positioning thread is consistent with the current version number. If they are inconsistent, the backend positioning thread interrupts the positioning process based on the recorded camera list, updates the recorded camera list and the recorded version number based on the normal camera list, and returns to re-determine whether the recorded version number is consistent with the current version number. If they are consistent, the number of normal cameras is determined based on the recorded camera list.

8. The method according to claim 1, characterized in that, The method further includes: In response to detecting a cold start of the multi-camera positioning system, determine whether each camera in the multi-camera positioning system is malfunctioning; If there are no faulty cameras in the multi-camera positioning system, the multi-camera positioning system is controlled to operate in multi-camera mode. If there are faulty cameras in the multi-camera positioning system, the system returns to re-determine whether each camera in the multi-camera positioning system is faulty, until the number of determinations reaches a preset threshold or the determination time reaches a preset time threshold, and then an early warning message is generated and displayed.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images