Control method of movable device, movable device, electronic device, and medium

Abstract

The application provides a control method of a movable device, the movable device, an electronic device and a medium. The method acquires a first image set and a second image set of an environment in which the movable device is currently located. A first image in the first image set is an environment image of a front field of view of the movable device, and a second image in the second image set is an environment image of a rear field of view of the movable device. Each first image is input into a front perception model to obtain a first detection result through target detection. Each second image is input into a rear perception model to obtain a second detection result. If it is monitored that one of the front perception model and the rear perception model is in an invalid working state, a current control mode of the movable device is adjusted to a degraded control mode. A first control instruction is generated based on the degraded control mode and a detection result corresponding to the perception model in the valid working state to control the movable device to move. The embodiments of the application can improve the safety of the movable device.

Description

Technical Field

[0001] This application relates to the field of equipment control technology, and more specifically, to a control method for a mobile device, a mobile device, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of autonomous driving technology, bird's eye view (BEV) perception technology based on multi-view cameras has become a hot research topic in the industry because it can provide a unified spatial representation. Typical BEV algorithms generally adopt a "panoramic unified processing" architecture: that is, by using multiple cameras arranged around the vehicle (usually including six fields of view such as front view, side front, side rear, and rear view), the collected image data is input into a unified deep neural network. Multi-view features are extracted using a shared backbone network, and the features are mapped to a unified bird's eye view space for fusion, ultimately realizing target detection and drivable area identification in the surrounding environment.

[0003] However, with the above method, if the shared module fails due to hardware failure, software anomaly, loss of input data, or disconnection of the link, the entire perception system will collapse and be unable to output any effective environmental information, which greatly weakens the robustness of the system and makes it difficult to meet the strict safety requirements of autonomous driving systems. Summary of the Invention

[0004] In view of this, this application provides a control method for a mobile device, a mobile device, an electronic device, and a storage medium to at least solve the problems existing in the related art.

[0005] Specifically, this application is implemented through the following technical solution: This application provides a control method for a mobile device, including: A first image set and a second image set of the environment in which the mobile device is currently located are acquired. The first image set includes multiple first images, which are environmental images of the front field of view of the mobile device. The second image set includes multiple second images, which are environmental images of the rear field of view of the mobile device. The front is consistent with the forward direction of the mobile device. Each of the first images is input into the forward-looking perception model for target detection to obtain a first detection result, and each of the second images is input into the backward-looking perception model to obtain a second detection result; Monitor whether the forward-looking perception model and the rear-looking perception model are in an effective working state, and if the monitoring result indicates that one of the forward-looking perception model and the rear-looking perception model is in an ineffective working state, adjust the current control mode of the mobile device to a degraded control mode. Based on the degradation control mode and the detection results corresponding to the perception model in an effective working state, a first control command is generated, and the mobile device is controlled to move according to the first control command.

[0006] This application also provides a mobile device, including a controller, the controller comprising: The memory is configured to store instructions; and The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the steps of the control method for the mobile device described in the foregoing embodiments.

[0007] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the control method for any of the foregoing embodiments of the mobile device.

[0008] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the control method for any of the foregoing embodiments of the mobile device.

[0009] This application also provides a computer program product, including a computer program that, when run by a processor, performs the steps of any of the possible control methods for mobile devices described above.

[0010] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this embodiment, the processing mechanism of separation (processing the first image set and the second image set separately), monitoring (monitoring the state of the perception model), and degradation (degradation control when any perception model fails) can avoid the problem of "system crash due to single perception model failure" in related technologies. The first control command is generated by the degradation control mode and the detection results corresponding to the perception model in an effective working state, thereby improving the mobile security of the mobile device.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a control method for a mobile device according to an exemplary embodiment of this application; Figure 2 This is a logical schematic diagram illustrating the working state monitoring of a perception model according to an exemplary embodiment of this application; Figure 3This is an exemplary embodiment of the present application illustrating a mode selection flowchart corresponding to the failure of any perception model; Figure 4 This is a flowchart illustrating another control method for a mobile device according to an exemplary embodiment of this application; Figure 5 This is a logical flowchart illustrating a working condition determination according to an exemplary embodiment of this application; Figure 6 This is a schematic diagram of a control architecture based on a dual-sensor model, as illustrated in an exemplary embodiment of this application; Figure 7 This is a hardware structure diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0013] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0014] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0015] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0016] With the rapid development of autonomous driving technology, bird's eye view (BEV) perception technology based on multi-view cameras has become a hot research topic in the industry because it can provide a unified spatial representation. Typical BEV algorithms generally adopt a "panoramic unified processing" architecture: that is, by using multiple cameras arranged around the vehicle (usually including six fields of view such as front view, side front, side rear, and rear view), the collected image data is input into a unified deep neural network. Multi-view features are extracted using a shared backbone network, and the features are mapped to a unified bird's eye view space for fusion, ultimately realizing target detection and drivable area identification in the surrounding environment.

[0017] However, despite their good perception performance under ideal conditions, these methods still face the following significant challenges in practical vehicle deployments, limiting their widespread application in scenarios with high reliability and real-time requirements: 1. Inefficient allocation of computing power and low resource utilization. Current BEV models generally employ a "one-size-fits-all" processing strategy, performing feature extraction and fusion calculations with equal intensity on image data from the front, rear, and side views. However, real-world driving behavior exhibits significant differences in directional focus. In most conventional driving scenarios (such as straight-line cruising and following other vehicles), the driver's attention is highly concentrated on the forward field of view in the direction of vehicle travel to address critical risks such as sudden braking by the vehicle in front or the appearance of obstacles; while the rear and rear-side views only require basic monitoring to determine the feasibility of lane changes or the status of approaching vehicles behind. Existing technologies fail to simulate the cognitive mechanisms of human driving, leading to excessive computational resources being allocated to non-critical areas (such as the rear field of view), resulting in a serious waste of computing resources and hindering the achievement of efficient perception through dynamic allocation on demand.

[0018] 2. The system has high coupling, posing a risk of single point of failure. Current mainstream BEV architectures generally rely on a shared backbone network to process all field-of-view images, with the front and rear view perception paths deeply coupled during the feature extraction stage. If this shared module fails due to hardware malfunction, software anomaly, or input data loss, the entire perception system will collapse, unable to output any valid environmental information. This "all-or-nothing" perception mode significantly weakens the system's robustness, making it difficult to meet the stringent safety requirements of autonomous driving systems for fail-operational (operability under fault) and fail-safe (safe shutdown under fault).

[0019] 3. Weak fault response capabilities and lack of a tiered and degraded fault mechanism. Existing technologies lack flexible response strategies when partial failures occur in the perception subsystem. For example, when forward perception fails due to occlusion, strong light interference, or abnormal model inference, the system often cannot effectively utilize the still-normal rear-view perception information, causing the control strategy to degenerate directly into the most conservative emergency braking or stopping, affecting driving continuity and user experience. Furthermore, existing solutions generally lack mechanisms to dynamically adjust the control mode based on the effective perception area, failing to construct a layered defense system of "partial perception – partial control," and making it difficult to achieve true functional safety degradation.

[0020] 4. The control strategy is disconnected from the perceived state, lacking closed-loop linkage. Most systems design the sensing and control modules separately, processing all sensing results, whether complete or incomplete, using the same set of control logic. This fails to dynamically adjust control behavior based on the health status and information integrity of the sensing system. This static mapping relationship is prone to misjudgment or overreaction under complex or abnormal operating conditions, making it impossible to achieve true "sensing-decision-control" collaborative optimization.

[0021] Based on the above research, this disclosure provides a control method for a mobile device. The method first acquires a first image set and a second image set of the environment currently in which the mobile device is located. The first image set includes multiple first images, which refer to environmental images of the front field of view of the mobile device. The second image set includes multiple second images, which refer to environmental images of the rear field of view of the mobile device; wherein, the front is consistent with the forward direction of the mobile device. Then, each first image is input into a forward-looking perception model for target detection to obtain a first detection result, and each second image is input into a rear-looking perception model to obtain a second detection result. Simultaneously, the method monitors whether the forward-looking perception model and the rear-looking perception model are in an effective working state. If the monitoring result indicates that either the forward-looking perception model or the rear-looking perception model is in an ineffective working state, the current control mode of the mobile device is adjusted to a degraded control mode. Finally, based on the degraded control mode and the detection results corresponding to the perception model in an effective working state, a first control command is generated, and the mobile device is controlled to move according to the first control command.

[0022] In this embodiment, the processing mechanism of separation (processing the first image set and the second image set separately), monitoring (monitoring the state of the perception model), and degradation (degradation control when any perception model fails) can avoid the problem of "system crash due to single perception model failure" in related technologies. The first control command is generated by the degradation control mode and the detection results corresponding to the perception model in an effective working state, thereby improving the mobile security of the mobile device.

[0023] To facilitate understanding of this embodiment, the executing entity of the control method for a mobile device provided in this disclosure will first be described in detail. The executing entity of the control method for a mobile device provided in this disclosure is generally a mobile device, which can be a vehicle, robot, etc., and is not limited thereto.

[0024] In some embodiments, the executing entity of the method can also be an electronic device, which can be a server. This server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In other embodiments, the electronic device can also be a terminal device, which can be a mobile device, terminal, handheld device, computing device, vehicle-mounted device, wearable device, etc.

[0025] In other embodiments, the method can also be applied to an implementation environment consisting of a mobile device and a server. Furthermore, the control method for the mobile device can also be implemented by a processor calling computer-readable instructions stored in memory.

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] Please see the appendix Figure 1 The above is a flowchart illustrating a control method for a mobile device, as shown in an exemplary embodiment of this application. Figure 1 As shown, the control method for the mobile device in this embodiment may include the following steps S101~S104: S101: Obtain a first image and a second image of the environment currently in which the mobile device is located. The first image refers to the environmental image of the front field of view of the mobile device, and the second image refers to the environmental image of the rear field of view of the mobile device; wherein, the front is consistent with the forward direction of the mobile device.

[0028] The mobile device refers to a mobile device with autonomous driving capabilities. In this embodiment, the mobile device is an autonomous vehicle, which includes, but is not limited to, cars, buses, trucks, and tractor-trailers. In other embodiments, the mobile device can also be a robot, which is also not limited. The robot can be a cargo robot, patrol robot, welcoming robot, etc. This embodiment uses a vehicle as an example for explanation.

[0029] Here, the first image set includes multiple first images. The first image refers to the environmental image of the front field of view (including the front, left front, and right front) of the mobile device. The front is consistent with the forward direction of the mobile device. The front being consistent with the forward direction of the mobile device can mean that the angle (absolute value) between the front and the forward direction is less than a preset angle. The preset angle can be set according to actual needs, such as 30 degrees, 60 degrees, or 90 degrees, and is not limited here.

[0030] Specifically, the multiple first images can be environmental images captured by multiple first image acquisition devices located at the front of the mobile device (including the front, left front, and right front). The multiple first image acquisition devices correspond to different shooting angles, and the coverage of the shooting angles of the multiple first image acquisition devices is the range of the front field of view.

[0031] Similarly, the second image set includes multiple second images, which refer to the environmental images of the rear field of view (including the front rear, left rear, and right rear) of the mobile device. Specifically, the multiple second images can be environmental images captured by multiple second image acquisition devices located at the rear of the mobile device (including the front rear, left rear, and right rear). The multiple second image acquisition devices correspond to different shooting angles, and the coverage of the shooting angles of the multiple second image acquisition devices is the range of the rear field of view.

[0032] Specifically, both the first image acquisition device and the second image acquisition device refer to cameras. The cameras may include fisheye cameras, depth cameras, infrared cameras, RGB sensors, etc., and are not limited here.

[0033] It should be understood that the coverage of the different shooting angles corresponding to the multiple first image acquisition devices and the multiple second image acquisition devices is 360 degrees, so that environmental information around the mobile device can be collected.

[0034] S102: Input each of the first images into the forward-looking perception model to perform target detection and obtain a first detection result, and input each of the second images into the backward-looking perception model to obtain a second detection result.

[0035] It is understandable that both forward-looking perception models and backward-looking perception models refer to deep learning neural networks deployed on mobile devices.

[0036] Here, the forward perception model and the rearward perception model run independently on different hardware architectures.

[0037] The first detection result and the second detection result include information such as target bounding box (or target detection box), target category, and confidence level. Of course, in other embodiments, orientation information may also be included, which is not limited here.

[0038] S103: Monitor whether the forward-looking perception model and the rear-looking perception model are in an effective working state, and if the monitoring result indicates that one of the forward-looking perception model and the rear-looking perception model is in an invalid working state, adjust the current control mode of the mobile device to a degraded control mode.

[0039] The current control mode of the mobile device can be either cruise mode or non-cruise mode. Degraded control mode refers to the operating mode under the safety protection mechanism, including emergency avoidance mode and lane keeping mode.

[0040] It should be understood that in actual operating scenarios, the forward perception model itself, the backward perception model itself, the data transmission link corresponding to the forward perception model, the data transmission link corresponding to the backward perception model, the hardware connection, program and computing resources corresponding to the two models may all be abnormal, resulting in the forward perception model and the backward perception model being in an invalid working state.

[0041] If either the forward-looking perception model or the rear-looking perception model is detected to be in an invalid state, the current control mode of the mobile device will be adjusted to a degraded control mode to ensure safety when perception capabilities are limited.

[0042] In this embodiment, monitoring whether the forward-looking perception model and the rear-looking perception model are in an effective working state can mean that if the survival information of the forward-looking model of the forward-looking perception model is detected within a first preset time period, then the forward-looking perception model is determined to be in an effective working state. Similarly, if the survival information of the rear-looking model of the rear-looking perception model is detected within a second preset time period, then the rear-looking perception model is determined to be in an effective working state.

[0043] Specifically, in this embodiment, the forward-looking perception model is pre-configured to periodically generate forward-looking model survival information according to a first preset time interval when it is in an effective working state, so as to indicate that it is in an effective working state. In this way, the controller in the mobile device will monitor the forward-looking model survival information in real time. If the forward-looking model survival information is detected within the first preset time interval, it can be determined that the forward-looking perception model is in an effective working state. Conversely, if the forward-looking model survival information is not detected within the first preset time interval, it can be determined that the forward-looking perception model is in a failed working state.

[0044] Similarly, the rear-view perception model is pre-configured to periodically generate front-view model survival information at a second preset time interval when it is in an effective working state, in order to indicate that it is in an effective working state. In this way, the controller in the mobile device will monitor the rear-view model survival information in real time. If the rear-view model survival information is detected within the second preset time interval, it can be determined that the rear-view perception model is in an effective working state. Conversely, if the rear-view model survival information is not detected within the second preset time interval, it can be determined that the rear-view perception model is in a failed working state.

[0045] The first preset time interval can refer to the time required for the forward-looking perception model to perform object detection on a first frame of the first image. Similarly, the second preset time interval can refer to the time required for the backward-looking perception model to perform object detection on a second frame of the second image. The first and second preset time intervals can be the same or different, and this is not limited here. It should be noted that since the first and second preset time intervals are preset, it should be understood that the time taken by the forward-looking perception model and the backward-looking perception model to perform object detection on different images is relatively similar. Therefore, the upper limit of the object detection time (i.e., the first and second preset time intervals) can be set according to the actual situation.

[0046] That is, the forward-looking perception model generates forward-looking model survival information after each frame of the first image is detected, and the backward-looking perception model generates backward-looking model survival information after each frame of the second image is detected.

[0047] In practice, the controller maintains a watchdog timer and continuously monitors the survival information of the forward-looking perception model and the rear-looking perception model. If the corresponding preset time interval is exceeded, the corresponding perception model is immediately determined to be faulty, and the subsequent degraded control mode is triggered.

[0048] Please see Figure 2 This is a logical schematic diagram of a perception model for monitoring its working state, provided as an exemplary embodiment of this application. Figure 2 As shown, the forward-looking perception model and the rear-looking perception model send survival information to the controller. The watchdog timer in the controller determines whether the preset time interval has been exceeded. If the preset time interval has not been exceeded, monitoring continues. If the preset time interval has been exceeded, the degraded control mode is triggered.

[0049] S104: Based on the degradation control mode and the detection results corresponding to the perception model in an effective working state, generate a first control command, and control the mobile device to move according to the first control command.

[0050] The first control command is the control information for the mobile device, such as throttle control information, brake control information, steering wheel angle control information, etc.

[0051] Here, in the event that any perception model fails, a first control command is generated based on the degraded control mode and the detection results corresponding to the perception model in a valid working state to control the movement of the mobile device.

[0052] In this embodiment, the processing mechanism of separation (processing the first image set and the second image set separately), monitoring (monitoring the state of the perception model), and degradation (degradation control when any perception model fails) can avoid the problem of "system crash due to single perception model failure" in related technologies. The first control command is generated by the degradation control mode and the detection results corresponding to the perception model in an effective working state, thereby improving the mobile security of the mobile device.

[0053] Specifically, by decoupling the forward-looking perception model from the rear-looking perception model, the physical and logical isolation of the perception path is achieved. When a branch of the perception model fails due to sensor malfunction, algorithm abnormality, or environmental interference (such as strong light or occlusion) or link interruption, the system can promptly monitor and trigger a degraded control mode, avoiding the collapse of the entire system due to local failure. This effectively solves the technical pain point of "single-point failure causing global paralysis" in traditional BEV architecture, and significantly enhances the fail-safe and fail-operational capabilities of the autonomous driving system.

[0054] Secondly, using an independent dual-sensor model to process front and rear view images avoids redundant processing of non-critical fields of view that is costly and computationally expensive. The computing power allocation strategy can be dynamically adjusted according to the driving scenario (such as high-precision inference for front view and lightweight model for rear view), simulating the attention mechanism of human drivers, achieving "on-demand perception", and improving the overall energy efficiency ratio.

[0055] Finally, a dynamic control mechanism based on effective perception state is introduced, so that the generation of control commands no longer depends on "full perception results", but makes adaptive decisions based on the currently available perception information. In degraded mode, the system can still use the detection results of effective directions (such as controlling the vehicle to stop safely when only the rear view is available) to achieve limited but safe control behavior, avoid secondary risks caused by "blind driving" or sudden stops, and support the construction of layered and progressive defensive driving strategies.

[0056] As mentioned above, the downgraded control mode includes emergency avoidance mode and lane keeping mode. Therefore, for step S103, if the forward perception model is invalid, the emergency avoidance mode is used to control the mobile device. Specifically, it may include the following steps (1) to (2): (1) If the monitoring results indicate that the forward perception model is in an invalid working state, the current control mode of the mobile device is adjusted to the emergency avoidance mode.

[0057] (2) Based on the emergency avoidance mode and the second detection result corresponding to the rear-view perception model, the control command is generated.

[0058] Here, due to the failure of the forward-looking perception model, environmental information regarding the vehicle's direction of travel cannot be obtained from the first image, thus necessitating emergency avoidance.

[0059] The emergency avoidance mode includes a directional control strategy, a deceleration strategy, and a lane-change stopping strategy. The directional control strategy can refer to maintaining short-term heading stability based on the vehicle's inertial navigation information. The deceleration strategy can be executed with the assistance of other types of sensor data (such as microwave or ultrasonic ranging radar) to maintain the distance between the mobile device and the vehicle in front. The lane-change stopping strategy can be executed based on the second detection result of the rear-view perception model.

[0060] In this way, by using the emergency avoidance mode and the second detection result corresponding to the rear-view perception model, the vehicle can maintain a short-term directional stability by using the second detection result in conjunction with inertial navigation information, without causing a collision within a safe distance. This allows the vehicle to decelerate smoothly and change lanes to a safe area with no fast-moving vehicles behind it. In this way, the risk of rear-end collisions can be avoided to the greatest extent possible when the forward-view perception model fails.

[0061] Similarly, regarding step S103, if the rear-view perception model is invalid, lane keeping mode is used to control the mobile device. Specifically, this may include the following steps (I) to (II): (I) If the monitoring results indicate that the rear-view perception model is in an invalid working state, the current control mode of the mobile device is adjusted to the lane keeping mode.

[0062] (II) Based on the lane keeping mode and the first detection result corresponding to the forward perception model, the control command is generated.

[0063] The lane keeping mode includes lane keeping strategy and following distance increase strategy.

[0064] Here, if the rear-view perception model is in an invalid working state, it means that environmental information behind the mobile device cannot be obtained. Therefore, the current control mode of the mobile device needs to be adjusted to lane keeping mode. That is, according to the first detection results, the mobile device is controlled to continue driving along the current lane and cannot change lanes. At the same time, the adaptive cruise strategy is used to increase the following distance until the driver takes over.

[0065] Please see Figure 3 This is a flowchart illustrating the mode selection process corresponding to the failure of any perception model, as provided in an exemplary embodiment of this application. Figure 3 As shown, when the controller detects that the forward-looking perception model is in a malfunctioning state, it adjusts the current control mode of the mobile device to the emergency avoidance mode, thereby executing the direction control strategy (inertial navigation maintains heading), the deceleration strategy (controls vehicle deceleration), and the lane-changing and stopping strategy (changes lanes to a safe area and stops). When the controller detects that the rear-looking perception model is in a malfunctioning state, it adjusts the current control mode of the mobile device to the lane-keeping mode, thereby executing the lane-keeping strategy (locks the current lane and drives) and the following distance increase strategy (increases the following distance and waits for the driver to take over).

[0066] Please see Figure 4 This is a flowchart illustrating another control method for a mobile device provided in an exemplary embodiment of this application. Figure 4 As shown, steps S401 to S408 are included: S401: Obtain a first image set and a second image set of the environment currently in which the mobile device is located. The first image set includes multiple first images, which refer to the environmental images of the front field of view of the mobile device. The second image set includes multiple second images, which refer to the environmental images of the rear field of view of the mobile device. The front is consistent with the forward direction of the mobile device.

[0067] The content of step S401 is similar to that of step S101 in the previous text. For details, please refer to step S101, which will not be repeated here.

[0068] S402: Input each of the first images into the forward-looking perception model to perform target detection and obtain a first detection result, and input each of the second images into the backward-looking perception model to obtain a second detection result.

[0069] The content of step S402 is similar to that of step S102 in the previous text. For details, please refer to step S102, which will not be repeated here.

[0070] S403: Monitor whether the forward perception model and the rear perception model are in a working state. If yes, proceed to step S404; if no, proceed to step S407.

[0071] The content of step S403 is similar to that of step S103 in the previous text. For details, please refer to step S103, which will not be repeated here.

[0072] S404: Determine whether the current control mode of the mobile device is cruise mode. If yes, proceed to step S405; otherwise, proceed to step S406.

[0073] Here, the current control mode can be determined as cruise mode based on the current control information of the mobile device. Specifically, the current control information includes turn signal, steering wheel angle signal, and planning information (autonomous driving planning information, such as the planned path).

[0074] S405: The first detection results corresponding to the multiple first images are fused with the second detection results corresponding to the multiple second images to obtain a fused detection result. Based on the fused detection result, a first bird's-eye view image reflecting the environment of the mobile device is generated. Based on the first bird's-eye view image, a second control command is generated, and the mobile device is controlled to move according to the second control command.

[0075] If the current control mode of the mobile device is cruise mode, in order to maximize computational efficiency and reduce system latency, a result-level fusion strategy is adopted, that is, the first detection result and the second detection result are fused to obtain the fused detection result.

[0076] Cruise mode refers to the cruise mode of a mobile device along the current lane.

[0077] Specifically, when fusing the first detection results with the second detection results, the first detection results and the second detection results can be transformed into the coordinate system of the mobile device itself. For redundant targets appearing in the overlapping area, the intersection-union ratio is calculated for matching, and non-maximum suppression or a weighted average algorithm based on sensor confidence is used for deduplication and stitching to obtain the fused detection result. After obtaining the fused detection result, a first bird's-eye view image reflecting the panoramic environment around the mobile device can be generated based on the fused detection result. Furthermore, a second control command is generated based on the first bird's-eye view image, and the mobile device is controlled to move according to the second control command.

[0078] In this embodiment of the application, the above-mentioned result-level fusion strategy is adopted in cruise mode, which can improve fusion efficiency and reduce system latency.

[0079] Optionally, when fusing each first detection result with each second detection result to obtain a fused detection result, since the cruise mode requires more attention to the environment in front of (and to the side front of) the mobile device, the weight of each first detection result can be increased when fusing the detection results, thereby improving the detection accuracy of the environment in front of and to the side front.

[0080] S406: The first image features corresponding to the multiple first images are fused with the second image features corresponding to the multiple second images to obtain fused image features. Based on the fused image features, a second bird's-eye view image reflecting the environment of the mobile device is generated. A third control command is generated based on the second bird's-eye view image, and the mobile device is controlled to move according to the third control command.

[0081] If the current control mode of the mobile device is non-cruise mode, the first image features corresponding to each first image and the second image features corresponding to each second image are fused to obtain fused image features. Based on the fused image features, a second bird's-eye view image reflecting the panoramic environment of the mobile device is generated to generate a third control command, and the mobile device is controlled to move according to the third control command.

[0082] Non-cruise mode can include modes for complex operating conditions such as lane changing and steering.

[0083] In this step, under non-cruise mode, the aforementioned feature-level fusion strategy can improve the spatial continuity and fusion accuracy of environmental perception, thereby improving the control accuracy of the third control command.

[0084] Optionally, when fusing the first image features with the second image features to obtain the fused image features, since lane changing, turning and other conditions require more attention to the environment behind and to the side of the mobile device, the weight of the second image features can be increased during feature fusion, thereby improving the detection accuracy of the environment behind and to the side.

[0085] Please see Figure 5 This is a logic flowchart for determining operating conditions, provided as an exemplary embodiment of this application. Figure 5 As shown, the controller obtains the current control information of the mobile device (including turn signal information, steering wheel angle information, and planning information) through the CAN bus. Based on the current control information, it determines whether it is in cruise mode. If so, it performs result-level fusion (fusion of each first detection result with each second detection result). If it is not in cruise mode, it performs feature-level fusion (fusion of each first image feature with each second image feature).

[0086] S407: If the monitoring results indicate that one of the forward-looking perception model and the rear-looking perception model is in an invalid working state, the current control mode of the mobile device is adjusted to a degraded control mode.

[0087] The content of step S407 is similar to that of step S103 described above, and will not be repeated here.

[0088] S408: Based on the degraded control mode and the detection results corresponding to the perception model in an effective working state, generate a first control command, and control the mobile device to move according to the first control command.

[0089] The content of step S407 is similar to that of step S104 described above, and will not be repeated here.

[0090] Please see Figure 6 This is a schematic diagram of a control architecture based on a dual-sensor model, provided as an exemplary embodiment of this application. Figure 6 As shown, the forward-looking camera group acquires a first image set and sends it to the forward-looking perception model for target detection, obtaining the first detection result (and first image features) corresponding to each first image. The rear-looking camera group acquires a second image set and sends it to the rear-looking perception model for target detection, obtaining the second detection result (and second image features) corresponding to each second image. The controller continuously monitors whether the forward-looking perception model and the rear-looking perception model are in an effective working state. If the monitoring result indicates that both the forward-looking perception model and the rear-looking perception model are in an effective working state, the controller determines the current control mode (operating condition) based on the current control information and selects the corresponding fusion strategy according to the operating condition, thereby fusing the detection results (or image features) output by the two perception models to obtain the fusion result for subsequent vehicle control. If the monitoring result indicates that the forward-looking perception model is in an effective working state, degraded control is performed based on the first detection result of the forward-looking perception model. If the monitoring result indicates that the rear-looking perception model is in an effective working state, degraded control is performed based on the second monitoring result of the rear-looking perception model. That is, the controller will select the corresponding degraded control mode for degraded control.

[0091] Corresponding to the control method for the aforementioned mobile device, this disclosure also provides an electronic device, such as... Figure 7 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Figure 7 As shown, the electronic device 700 includes a processor 710, an internal bus 720, a memory 730, a network interface 740, and a non-volatile memory 750, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 710 reads the corresponding computer program from the non-volatile memory 750 into the memory 730 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0092] The memory 730, also known as internal memory, is used to temporarily store the computational data in the processor 710, as well as the data exchanged with non-volatile memory 750 such as hard disk. The processor 710 exchanges data with the non-volatile memory 750 through the memory 730.

[0093] In this embodiment, memory 730 is specifically used to store application code that executes the solution of this application, and its execution is controlled by processor 710. That is, when the electronic device is running, processor 710 communicates with network interface 740, memory 730 and non-volatile memory 750 through internal bus 720, so that processor 710 executes the application code stored in memory 730 and non-volatile memory 750, thereby executing the control method of the mobile device described in the above method embodiment.

[0094] Processor 710 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware microservices. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0095] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 700. In other embodiments of this application, the electronic device 700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0096] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the control method for the mobile device described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0097] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the control method for the mobile device in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0098] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0099] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0100] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0101] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. Basic computer microservices include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0102] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0103] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0104] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and microservices in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program microservices and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0105] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0106] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A control method for a mobile device, characterized in that, include: A first image set and a second image set of the environment in which the mobile device is currently located are acquired. The first image set includes multiple first images, which are environmental images of the front field of view of the mobile device. The second image set includes multiple second images, which are environmental images of the rear field of view of the mobile device. The front is consistent with the forward direction of the mobile device. Each of the first images is input into the forward-looking perception model for target detection to obtain a first detection result, and each of the second images is input into the backward-looking perception model to obtain a second detection result; Monitor whether the forward-looking perception model and the rear-looking perception model are in an effective working state, and if the monitoring result indicates that one of the forward-looking perception model and the rear-looking perception model is in an ineffective working state, adjust the current control mode of the mobile device to a degraded control mode. Based on the degradation control mode and the detection results corresponding to the perception model in an effective working state, a first control command is generated, and the mobile device is controlled to move according to the first control command.

2. The method according to claim 1, characterized in that, The monitoring of whether the forward-looking perception model and the backward-looking perception model are in a valid working state includes: If the forward-looking model survival information of the forward-looking perception model is detected at a first preset time interval, it is determined that the forward-looking perception model is in a valid working state; the forward-looking model survival information is generated periodically by the forward-looking perception model according to the first preset time interval; and / or, If the rearview model survival information of the rearview perception model is detected within the second preset time interval, it is determined that the rearview perception model is in an effective working state; the rearview model survival information is generated periodically by the rearview perception model according to the second preset time interval.

3. The method according to claim 1, characterized in that, The degraded control mode includes an emergency avoidance mode; the step of adjusting the current control mode of the mobile device to the degraded control mode when the monitoring results indicate that one of the forward-looking perception model and the rear-looking perception model is in an invalid working state includes: If the monitoring results indicate that the forward-looking perception model is in an ineffective working state, the current control mode of the mobile device is adjusted to the emergency avoidance mode; Based on the degraded control mode and the detection results corresponding to the perception model in an effective working state, control commands are generated, including: The control command is generated based on the emergency avoidance mode and the second detection result corresponding to the rear-view perception model.

4. The method according to claim 3, characterized in that, The emergency avoidance mode includes at least one of the following: directional control strategy, deceleration strategy, and lane change and stop strategy.

5. The method according to claim 1, characterized in that, The degraded control mode includes a lane-keeping mode; the step of adjusting the current control mode of the mobile device to a degraded control mode when the monitoring results indicate that one of the forward-looking perception model and the rear-looking perception model is in an ineffective state includes: If the monitoring results indicate that the rear-view perception model is in an invalid working state, the current control mode of the mobile device is adjusted to the lane keeping mode; Based on the degraded control mode and the detection results corresponding to the perception model in an effective working state, control commands are generated, including: The control command is generated based on the lane keeping mode and the first detection result corresponding to the forward perception model.

6. The method according to claim 5, characterized in that, The lane keeping mode includes lane keeping strategy and following distance increase strategy.

7. The method according to claim 1, characterized in that, The method further includes: When the monitoring results indicate that both the forward-looking perception model and the rear-looking perception model are in an effective working state, if the current control mode of the mobile device is cruise mode, the first detection results corresponding to multiple first images and the second detection results corresponding to multiple second images are fused to obtain a fused detection result. Based on the fusion detection results, a first bird's-eye view image reflecting the environment in which the mobile device is located is generated, and a second control command is generated based on the first bird's-eye view image, and the mobile device is controlled to move according to the second control command.

8. The method according to claim 1, characterized in that, The forward-looking perception model includes a first feature extraction network, and the backward-looking perception model includes a second feature extraction network; the first feature extraction network is used to extract features from the first image to obtain first image features, and the second feature extraction network is used to extract features from the second image to obtain second image features; the method further includes: When the monitoring results indicate that both the forward-looking perception model and the rear-looking perception model are in an effective working state, if the current control mode of the mobile device is non-cruise mode, the first image features corresponding to multiple first images and the second image features corresponding to multiple second images are fused to obtain fused image features. Based on the fused image features, a second bird's-eye view feature is generated to reflect the environment in which the mobile device is located, and a third control command is generated based on the second bird's-eye view feature, and the mobile device is controlled to move according to the third control command.

9. A mobile device, characterized in that, Includes a controller, the controller comprising: The memory is configured to store instructions; and A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the control method of the mobile device according to any one of claims 1-8.

10. An electronic device, characterized in that, The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions that the processor can execute. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the control method of the mobile device as described in any one of claims 1-8 is performed.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the control method for the mobile device according to any one of claims 1-8.

Images