Perception system with attention module for processing visual data

Abstract

The perception system is adapted to receive visual data from a camera and includes a controller having a processor and a tangible non-transitory memory having instructions recorded thereon. A subsampling module, an object detection module, and an attention module are each selectively executable by the controller. The controller is configured to sample an input image from the visual data via the subsampling module to generate a rescaled full image frame. The controller is configured to extract feature data from the rescaled full image frame via the object detection module. Based on an output of the attention module, a region of interest in the rescaled full image frame is identified. The controller is configured to generate a first image based on the rescaled full image frame and a second image based on the region of interest, the second image having a higher resolution than the first image.

Description

Technical Field

[0001] This disclosure generally relates to a perception system for visual data. More specifically, this disclosure relates to a perception system having an attention module and an object detection module for processing visual data. Background Technology

[0002] Automated object detection methods are useful in many different settings, including navigation systems for robots and autonomous vehicles, surveillance equipment, and automated personal assistive devices. Challenges faced by automated object detection systems include handling variations within object categories and the diversity of visual images caused by lighting, surrounding scene, and object orientation. In addition to the complexity of visual data, another significant challenge is the amount of computational resources required to process the images, especially for applications requiring high-resolution sensing. Summary of the Invention

[0003] This document discloses a perception system for a camera. The perception system includes a controller having a processor and tangible, non-transitory memory on which instructions are recorded. A subsampling module, an object detection module, and an attention module are each selectively executable by the controller. Execution of instructions by the processor causes the controller to sample an input image from visual data via the subsampling module to generate a rescaled whole image frame. The controller is configured to extract feature data from the rescaled whole image frame via the object detection module. Based on the output of the attention module, a region of interest (ROI) is identified in the rescaled whole image frame. The controller is configured to generate a first image based on the rescaled whole image frame and a second image based on the ROI, the second image having a higher resolution than the first image.

[0004] In some embodiments, the camera is fixed to a vehicle, and the controller is configured to control the operation of the vehicle in part based on the first image and / or the second image. The object detection module may include a first backbone unit and a second backbone unit. The first backbone unit and the second backbone unit are adapted to extract feature data from the input image and the region of interest, respectively. In some embodiments, the controller is adapted to generate a second image based on a plurality of episode frames over time, each of the plurality of episode frames including feature data from the preceding episode frame.

[0005] A region of interest defines a center. The controller may be adapted to select a center from the region of potential centers based on the output of the attention module. In some embodiments, the output of the attention module includes an attention agent indicating a corresponding probability of a positive reward for each potential center within the region of the potential center. In other embodiments, the output of the attention module includes an attention agent indicating a corresponding predicted reward for each potential center within the region of the potential center.

[0006] The attention module may include a deep Q-value network trained to identify regions of interest. Training the deep Q-value network involves maximizing the reward. This reward can be obtained by comparing a first number of true positives obtained from the base-fact dataset with a second number of true positives obtained from the original dataset. Training the deep Q-value network may include: obtaining a corresponding loss as the difference between the reward and the predicted reward corresponding to each potential center within the region of the potential center; and minimizing the corresponding loss.

[0007] This paper discloses a method for operating a perception system having a camera that collects visual data and a controller having a processor and tangible non-transitory memory. The method includes: transmitting visual data from the camera to the controller. The controller is adapted to selectively execute a subsampling module, an object detection module, and an attention module. The method includes: sampling an input image from the visual data via the subsampling module to generate a rescaled entire image frame; and extracting feature data from the rescaled entire image frame via the object detection module. Based on the output of the attention module, a region of interest is identified in the rescaled entire image frame. The method includes: generating a first image based on the rescaled entire image frame and generating a second image based on the region of interest, the second image having a higher resolution than the first image.

[0008] This document also discloses a vehicle having a camera adapted to acquire visual data and a controller adapted to receive visual data from the camera, the controller having a processor and tangible non-transitory memory on which instructions are recorded. The controller is configured to sample an input image of the visual data via a subsampling module to generate a rescaled entire image frame. The controller is configured to extract feature data from the rescaled entire image frame via an object detection module. Regions of interest (ROIs) in the rescaled entire image frame are identified based on the output of an attention module. The controller is configured to generate a first image based on the rescaled entire image frame and a second image based on the ROI, the second image having a higher resolution than the first image.

[0009] This disclosure may also include the following schemes.

[0010] 1. A sensing system suitable for receiving visual data from a camera, the sensing system comprising: A controller having a processor and tangible, non-transitory memory on which instructions are recorded; The sub-sampling module, object detection module, and attention module can each be selectively executed by the controller; Wherein, the processor's execution of the instructions causes the controller to: The input image is sampled from the visual data via the sub-sampling module to generate a rescaled whole image frame; Feature data is extracted from the entire rescaled image frame via the object detection module; The region of interest in the entire rescaled image frame is identified, at least in part, based on the output of the attention module; and A first image is generated based on the rescaled entire image frame, and a second image is generated based on the region of interest, the second image having a higher resolution than the first image.

[0011] 2. The sensing system according to Scheme 1, wherein: The camera is fixed to the vehicle; and The controller is configured to control the operation of the vehicle in part based on the second image.

[0012] 3. The sensing system according to Scheme 1, wherein: The object detection module includes a first backbone unit and a second backbone unit; and The first backbone unit and the second backbone unit are adapted to extract feature data from the input image and the region of interest, respectively.

[0013] 4. The sensing system according to Scheme 1, wherein: The controller is adapted to generate a second image based on multiple episode frames over time, each of the multiple episode frames combining feature data from the preceding episode frame.

[0014] 5. The sensing system according to Scheme 1, wherein: The region of interest defines a center, and the controller is adapted to select a center from the region of potential centers based on the output of the attention module.

[0015] 6. The sensing system according to Scheme 5, wherein: The output of the attention module includes an attention agent indicating the probability of a positive reward corresponding to each potential center within the region of the potential center.

[0016] 7. The sensing system according to Scheme 6, wherein: The output of the attention module includes an attention agent indicating the corresponding predicted reward for each potential center within the region of the potential center.

[0017] 8. The sensing system according to Scheme 5, wherein: The attention module includes a deep Q-value network trained to identify regions of interest, and training the deep Q-value network includes maximizing the reward.

[0018] 9. The sensing system according to Scheme 8, wherein: Rewards are awarded by comparing a first number of true positives obtained from the basic fact dataset with a second number of true positives obtained from the original dataset.

[0019] 10. The sensing system according to Scheme 9, wherein: Training the deep Q-value network includes: obtaining a corresponding loss as the difference between the reward and the predicted reward corresponding to each potential center in the region of the potential center; and minimizing the corresponding loss.

[0020] 11. A method of operating a sensing system, the sensing system having a camera for collecting visual data and a controller having a processor and tangible non-transitory memory, the method comprising: Visual data is transmitted from the camera to the controller, which is adapted to selectively execute the sub-sampling module, the object detection module, and the attention module; The input image is sampled from the visual data via the sub-sampling module to generate a rescaled full image frame; Feature data is extracted from the entire rescaled image frame via the object detection module; The region of interest in the entire rescaled image frame is identified based on the output of the attention module; and A first image is generated based on the rescaled entire image frame, and a second image is generated based on the region of interest, the second image having a higher resolution than the first image.

[0021] 12. The method according to Scheme 11 further includes: Attach the camera to the vehicle; and The operation of the vehicle is controlled in part based on the second image via the controller.

[0022] 13. The method according to Scheme 11 further includes: The controller is adapted to generate the second image based on a plurality of episode frames over time, each of the plurality of episode frames incorporating feature data from the preceding episode frame.

[0023] 14. The method according to Scheme 11 further includes: A deep Q-value network is trained in the attention module to identify the region of interest, and training the deep Q-value network includes maximizing the reward.

[0024] 15. The method according to Scheme 14 further includes: Rewards are awarded by comparing a first number of true positives obtained from the basic fact dataset with a second number of true positives obtained from the original dataset.

[0025] 16. The method according to claim 11, wherein the region of interest defines a center, the method further comprising: The center is selected from the region of potential centers by the controller based on the output of the attention module.

[0026] 17. The method according to Scheme 16 further includes: An attention agent is generated via the attention module, which indicates the probability of a positive reward corresponding to each potential center within the region of the potential center.

[0027] 18. The method according to Scheme 16 further includes: An attention agent is generated via the attention module, which indicates the corresponding predicted reward for each potential center within the region of the potential centers.

[0028] 19. A vehicle comprising: A camera, which is suitable for acquiring visual data; A controller adapted to receive the visual data from the camera, the controller having a processor and tangible non-transitory memory on which instructions are recorded; The sub-sampling module, object detection module, and attention module can each be selectively executed by the controller; Wherein, the processor's execution of the instructions causes the controller to: The input image is sampled from the visual data via the sub-sampling module to generate a rescaled whole image frame; Feature data is extracted from the entire rescaled image frame via the object detection module; The region of interest in the entire rescaled image frame is identified based on the output of the attention module; Generate a second image of the region of interest, the second image having a higher resolution than the input image; and The operation of the vehicle is controlled in part based on the second image.

[0029] The foregoing features and advantages, as well as other features and advantages of this disclosure, will become apparent from the following detailed description of the best mode for carrying out this disclosure, taken in conjunction with the accompanying drawings. Attached Figure Description

[0030] Figure 1 It is a schematic fragment of a sensing system with a camera and a controller; Figure 2 It is discretized into a grid. Figure 1 A schematic example of the camera's field of view; Figure 3 It is an operation Figure 1 A flowchart of a method for a sensing system; Figure 4 It shows Figure 3 A schematic diagram of a part of the method; and Figure 5 It shows that it can be generated by Figure 1 A schematic diagram of an exemplary plot sampling sequence used by the controller.

[0031] Representative embodiments of this disclosure are shown by way of non-limiting example in the accompanying drawings and are described in further detail below. However, it should be understood that the novel aspects of this disclosure are not limited to the specific forms shown in the drawings listed above. Rather, this disclosure covers modifications, equivalents, combinations, sub-combinations, substitutions, groupings, and alternatives that fall within the scope of this disclosure, for example, as covered by the appended claims. Detailed Implementation

[0032] Referring to the accompanying drawings, the same reference numerals denote the same parts. Figure 1 A sensing system 10 with camera 12 is schematically shown. Camera 12 is rigidly fixed, connected, or attached (directly or indirectly) to platform 14. Platform 14 can be a non-mobile platform, such as a building structure, desktop computing device, or factory equipment. Platform 14 can also be a mobile platform, such as vehicle 16. Vehicle 16 can include, but is not limited to, buses, SUVs, light trucks, heavy vehicles, minivans, buses, transport vehicles, bicycles, mobile robots, agricultural implements (e.g., tractors), sports-related equipment (e.g., golf carts), ships, aircraft, and trains. It should be understood that vehicle 16 can take many different forms and have additional components.

[0033] refer to Figure 1 The sensing system 10 includes a controller C having at least one processor P and at least one memory M (or a non-transitory tangible computer-readable storage medium), wherein instructions for performing method 100 are recorded in the memory M, as described below. Figure 3 Detailed description. Memory M can store the controller executable instruction set, and processor P can execute the controller executable instruction set stored in memory M.

[0034] refer to Figure 1The perception system 10 includes multiple modules 18 that can be selectively executed by the controller C. The multiple modules 18 include a subsampling module 20, an object detection module 22, and an attention module 24. The perception system 10 reduces the number of pixels required to process the visual data acquired by the camera 12, for example, by using a lower resolution (e.g., standard resolution) for peripheral regions and by using a higher resolution (e.g., high resolution) to cover selected regions of interest. The object detection module 22 can be used to identify and track objects in the scene. For example, tracking objects may include drawing bounding boxes around each object of interest in the image and assigning them category labels. The object detection module 22 may include a region-based convolutional neural network (R-CNN).

[0035] Subsampling module 20 utilizes an attention agent generated by attention module 24 to extract information about the usefulness or importance of each portion of the visual data acquired by camera 12. The attention agent can be in the form of a matrix or other data store. Attention module 24 is based on reinforcement learning and is adapted, for example, by incorporating a deep Q-value network 26 to identify one or more regions of interest. In one example where platform 14 is a ground vehicle, attention module 24 can be trained / rewarded to find areas with small vehicles and select regions of interest to cover those areas. In another example where platform 14 is an aircraft, attention module 24 can be trained to guide regions of interest to cover the presence of birds.

[0036] Figure 1 The controller C can be an integral part of other controllers of vehicle 16, or a separate module operatively connected to other controllers. For example, controller C can be an electronic control unit (ECU) of vehicle 16. In some embodiments, controller C can be an integrated control unit embedded in camera 12. Controller C can receive input from other sensors (not shown) operatively connected to platform 14.

[0037] refer to Figure 1 The various components of the sensing system 10 can communicate with the controller C (and with each other) via a wireless network 28, which can be a short-range or long-range network. The wireless network 28 can be a communication bus, which may be in the form of a Serial Controller Area Network (CAN-BUS). The wireless network 28 may include Bluetooth connectivity, a wireless local area network (LAN) using a wireless distribution method to link multiple devices, a wireless metropolitan area network (MAN) connecting several wireless LANs, or a wireless wide area network (WAN). Other types of connections may also be used.

[0038] Figure 1The camera 12 is adapted to acquire visual information, referred to herein as visual data, and can be combined with various types of optical sensors and photodetectors available to those skilled in the art. The camera 12 may be a stereo camera adapted to acquire left and right image pairs. Figure 2 An exemplary field of view 50 of camera 12 is shown. The field of view 50 is shown to be discretized into a grid with a horizontal span 52 and a vertical span 54. In a non-limiting example, the horizontal span 52 and the vertical span 54 of the field of view 50 are 1280 pixels and 720 pixels, respectively.

[0039] The perception system 10 uses visual data from camera 12 (and multiple modules 18) to generate human-like attention signals for a selected region (here referred to as region of interest 60) within the field of view 50 of the visual data (see [reference]). Figure 2 High-resolution images are provided in the image. Generating high-resolution images by processing high-definition (HD) video streams requires significant computational resources. The perception system 10 (through the execution of method 100) generates a high-resolution image of the region of interest 60 without degrading performance and with a reduced number of image pixels processed. This allows the perception system 10 to be better mounted on embedded platforms and increases the utilization of multiple high-definition sensors. The perception system 10 is efficient in terms of computational and memory requirements and is independent because it is derived from perception and does not require prior map information.

[0040] refer to Figure 2 An example region of interest (ROI) 60 is shown. In the example shown, the horizontal grid span R1 of ROI 60 is 9 grid cells, while the vertical grid span R2 of ROI 60 is 5 grid cells. ROI 60 defines a center 62. Center 62 is selected from a set of potential centers 65 or regions 64. Reference Figure 2 Region 64 defines a first dimension 66 and a second dimension 68, which define the boundaries in which a potential center 65 may lie. Region 64 represents a subgroup of the grid of the field of view 50, which, based on the position and size of the region of interest 60, represents a possible center 62 for the region of interest 60. In other words, region 64 is selected such that the region of interest 60 (centered on one of the potential centers 65) does not fall outside the field of view 50.

[0041] Now for reference Figure 3 A flowchart of method 100 is shown. Method 100 can be embodied in a file stored in... Figure 1 Computer-readable code or instructions that are executed on and partially by the controller C. Method 100 may be executed in real time, continuously, systematically, sporadically, and / or at regular intervals, such as every 10 milliseconds during normal and ongoing operation of vehicle 16. Figure 3Method 100 begins at box 101 and ends at box 103, and includes subroutines or processing boxes 110, 120, 130, 140, 150, 160, 170, 180, and 190. Method 100 does not need to be applied in the specific order described herein. Furthermore, it should be understood that some boxes may be omitted.

[0042] according to Figure 3 In block 110, controller C is programmed to receive visual data from camera 12, including input image 70. The visual data can be a still image or a video stream. Moving to block 120, controller C is adapted to sample the input image 70 via subsampling module 20 to generate a rescaled whole image frame 72. Subsampling is a method of reducing data size by selecting a subset of the original data. Note that subsampling accelerates processing of large inputs by processing a portion of the input image data at high resolution.

[0043] Advance to Figure 3 In box 130, controller C is programmed to extract feature data from the rescaled entire image frame 72 (obtained in box 120) via object detection module 22. The feature data can be derived from feature tensor 210 (see feature tensor 210). Figure 3-4 Tensors are used to store, represent, or store data of different types. The values ​​of a tensor can maintain the same data type, i.e., a specific shape defining the dimensions of the matrix or array. In some embodiments, the object detection module 22 includes a convolutional neural network (CNN). (See also...) Figure 1 The object detection module 22 may include a first backbone unit 76, a first object detection head 78, a second backbone unit 80, and a second object detection head 82. The first backbone unit 76 is adapted to extract feature maps from the input image 70, which can be upsampled by the decoder model to generate segmented masks.

[0044] Method 100 can proceed from box 130 to both box 140 and box 150 simultaneously. According to... Figure 3 In box 140, the output of the first backbone unit 76 (e.g., feature tensor 210) is passed to the first object detection head 78. According to... Figure 3 In box 150, the output of the first backbone unit 76 (e.g., feature tensor 210) is passed to the attention module 24. (See reference) Figure 4 The feature tensor 210 includes multiple voxels 212 or three-dimensional data, which is "flattened" and fed into the first neural network 220 and the second neural network 230 of the attention module 24.

[0045] refer to Figure 4 The first neural network 220 includes an input layer 222, at least one hidden layer 224, and an output layer 226. The second neural network 230 includes an input layer 232, at least one hidden layer 234, and an output layer 236. (Reference) Figure 4 The first neural network 220 and the second neural network 230 can be adapted to obtain the first size 66 and the second size 68 of region 64, respectively (see Figure 2 (See above for reference) Figure 2 The controller C defines the position of the region of interest 60 within the field of view 50 by selecting the center 62 of the region of interest 60 from the region 64 of the potential center 65. Furthermore, according to... Figure 3 In frame 150, attention module 24 is selectively executed to generate attention agent 74 and identify at least one region of interest 60 in the rescaled entire image frame 72 based on attention agent 74. In some embodiments, at time t, attention agent 74 (A t ) is represented as: A t * = argmax A [(A t )).

[0046] refer to Figure 1 Attention module 24 may include a deep Q-value network 26 trained to identify regions of interest 60. The deep Q-value network 26 performs a series of actions that generate the maximum value of a reward function, called the Q-value. The deep Q-value network 26 predicts the attention agent 74 (A t The Q-value of the deep Q-value network 26. In some embodiments, the output of the deep Q-value network 26 can be represented as (A t =DQN (X) t-1 ,θ), where X t-1 It is a feature tensor (generated by object detection module 22), and θ represents the weights or parameters of the deep Q-value network 26 (DQN) (obtained during training). In other words, the deep Q-value network 26 estimates the Q-value of each possible action a based on the weights θ. The result is an attention agent 74, whose size corresponds to the region 64 of the latent center 65 (see...). Figure 2 The units are all the same size. That is, each action a is to select a potential center (in region 64) as the center 62 of region of interest 60.

[0047] Figure 3 Method 100 progresses from box 150 to box 160. Based on attention agent 74 (generated in box 150), subsampling module 20 generates window frame data for region of interest 60. The subsampling process can be represented as Subsample (I t A t =[ROI t W t ], where ROI t Is using Attention Agent 74 (A) t From input image I tThe image obtained from the selected window or region of interest is 60, and W t Indicates input image I t The entire image frame is rescaled.

[0048] Figure 3 Method 100 proceeds from box 160 to box 170. According to... Figure 3 In box 170, controller C is programmed to selectively execute object detection module 22 to extract feature data from window frame data (from box 160). More specifically, the second backbone unit 80 of object detection module 22 is adapted to extract features from the window frame data. Next, according to... Figure 3 The features extracted from the window frame data in frame 180 are transmitted to the second object detection head 82 within the object detection module 22 for further processing. Figure 3 Method 100 advances to box 190.

[0049] according to Figure 3 In box 190, as shown by line 185, data from the second object detection head 82 is used to generate a second image F2 based on the region of interest 60. As shown by line 145, data from the first object detection head 78 (from box 140) is used to generate a first image F1 based on the rescaled entire image frame 72. The second image F2 has a higher resolution than the first image F1. For example, the first image F1 can be a standard-resolution image of the (full) input image 70, and the second image F2 can be a high-resolution image of the region of interest 60.

[0050] The controller C can be programmed to control or influence the operation of the vehicle 16 using a first image F1 and a second image F2 from box 190. For example, operations such as adaptive cruise control, automatic braking, lane changing, and navigation systems can be modified based on corresponding analysis of the region of interest 60, the first image F1, and / or the second image F2.

[0051] Training a deep Q-value network 26 may include maximizing the reward. In some embodiments, the reward is obtained by comparing a first number of true positives obtained with the ground truth dataset with a second number of true positives obtained with the original dataset. The reward may be represented by a key performance indicator (KPI), which is an indication of the amount of progress made toward a measurable object. For example, the reward may be defined as R t = KPI [O t G t ], where G t It is the "basic fact" used for object detection, while O t It is a fusion of visual data used for the region of interest 60 and the entire image frame, for example, O t=Fusion [Object_Detector (ROI t ), Object_Detector (W t True positives refer to the number of detected objects that match the baseline fact. If TPB is the number of true positives before attention and TPA is the number of true positives after attention, then the reward (R) is... t The value of ) can be set as follows: (1) If TPA-TPB < 0, then R t = -1; (2) If TPA-TPB = 0, then R t =0; and (3) if TPA-TPB>0, then R t = 1.

[0052] Training the deep Q-value network 26 may include obtaining a loss as the difference between the actual reward and the predicted reward provided by the deep Q-value network 26. This loss is generated by comparing the predicted reward (output by the deep Q-value network 26) with the actual reward. The larger the difference between the predicted and actual rewards, the greater the loss. The training process attempts to minimize the loss in addition to maximizing the reward. As described above, the deep Q-value network 26 can be configured to output an attention agent 74 that indicates the corresponding predicted reward for a potential center 65 in region 64 (see [link to documentation]). Figure 3 In this context, determining the region of interest 60 includes: determining, based on the attention agent 74, which portion of region 64 is associated with the highest predicted reward. In some embodiments, the attention agent 74 indicates the corresponding probability of a positive reward for each potential center 65. Here, determining the region of interest 60 includes: determining which potential center 65 within region 64 is associated with the highest probability of generating a positive reward. The deep Q-value network 26 may implement additional algorithmic layers (e.g., softmax) available to those skilled in the art to obtain the probability of generating a positive reward.

[0053] In some embodiments, the controller C is adapted to generate a second image F2 based on a plurality of episode frames over time, wherein each of the plurality of episode frames incorporates feature data from the immediately preceding episode frame. In other words, the attention agent 74 in each episode t incorporates data from the feature tensor 210 of the preceding episode t-1. Figure 5 This is a schematic diagram illustrating an example of plot sampling that can be used by controller C. (Reference) Figure 5 The input image data 300 is fed into the first image matrix 302 (I1), which is used to generate the first complete image frame 304 (I... SD1The controller C is adapted to generate a first region of interest (ROI1) using a first image matrix 302 (I1) and a first feature tensor 306 (X1). Based on the input from the first attention agent 308 (A1), the controller C is adapted to generate a first region of interest (ROI1) using a first image matrix 302 (I1).

[0054] like Figure 5 As shown, data from the first feature tensor 306 (X1) and the first image matrix 302 (I1) are used to generate the second image matrix 312 (I2), which in turn generates the second complete image frame 314 (I1). SD2 The controller C generates a second image matrix 316 (X2) and a second feature tensor 316 (X2). Based on the input from the second attention agent 318 (A2), the controller C is adapted to generate a second region of interest 320 (ROI2) using the second image matrix 312 (I2). (See reference...) Figure 5 Data from the second feature tensor 316 (X2) is transferred to the next episode, namely the third attention agent 328 (A3). Therefore, in each episode t, the past feature tensor (X2) is retrieved from the replay buffer. t-1 In some embodiments, attention agent 74 can be used to pair policy πθ(A). t |X t-1 Sampling is performed, and the following is calculated: [ROI] t W t ] = Subsample(I t A t Here, ROI t Is it using agent A? t From input image I t The image obtained from the selected window or region of interest is 60, and W t Indicates input image I t The entire image frame is rescaled. Object detection module 22 can use ROI. t and W t Both operate, and key performance indicators can be evaluated as reward R. t Parameter (X) t-1 A t , R t The sampled data can be saved to the replay buffer. These sampling steps are repeated, and the object value y = R can be calculated. t +argmaxA (A t The loss function can be updated using stochastic gradient descent (SGD).

[0055] In summary, the perception system 10 (through the execution of method 100) uses visual data from camera 12 (and multiple modules 18) to generate human-like attention signals to provide a high-resolution image within the region of interest 60. By applying an attention subsampling strategy, including subsampling the input image resolution without degrading performance, the perception system 10 addresses the challenges of adapting perception routines to embedded platforms. The perception system 10 is self-contained and relatively easy to integrate into existing neural networks.

[0056] Figure 1 The controller C includes computer-readable media (also called processor-readable media), including non-transitory (e.g., tangible) media involved in providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Non-volatile media can include, for example, optical discs or magnetic disks and other permanent storage. Volatile media can include, for example, dynamic random access memory (DRAM), which can constitute main memory. These instructions can be transmitted by one or more transmission media, including coaxial cables, copper wires, and optical fibers, including cables containing a system bus coupled to the computer processor. Some forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tapes, other magnetic media, CD-ROMs, DVDs, other optical media, physical media with perforated patterns, RAM, PROMs, EPROMs, FLASH-EEPROMs, other memory chips or cartridges, or other media from which a computer can read them.

[0057] Lookup tables, databases, data repositories, or other data storage described herein may include various mechanisms for storing, accessing, and retrieving a variety of data, including hierarchical databases, filegroups in a file-based rechargeable energy storage system, application databases in proprietary formats, relational database energy management systems (RDBMS), etc. Each such data storage may be contained within a computing device employing one of the computer operating systems mentioned above and may be accessed via a network in one or more of various ways. File systems may be accessible from a computer-operated rechargeable energy storage system and may include files stored in various formats. In addition to languages ​​used for creating, storing, editing, and executing stored procedures, RDBMS may also employ Structured Query Language (SQL), such as the PL / SQL language described above.

[0058] The flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, including one or more executable instructions for implementing a specified logical function. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based rechargeable energy storage system or a combination of dedicated hardware and computer instructions that performs the specified function or action. These computer program instructions may also be stored in a computer-readable medium that can direct a controller or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of art including instructions for implementing the functions / actions specified in the blocks of the flowcharts and / or block diagrams.

[0059] In this specification (including the appended claims), the numerical values ​​of parameters (e.g., quantities or conditions) should be understood to be modified by the term "about" in each corresponding instance, regardless of whether "about" actually appears before the numerical value. "About" indicates that the numerical value allows for some slight imprecision (some exact closeness to the numerical value; approximately or reasonably close to the value; almost). If the imprecision provided by "about" cannot be understood in this ordinary sense in the art, then "about" as used herein at least indicates variations that may arise from common methods of measuring and using these parameters. Furthermore, the disclosure of ranges includes disclosing each value and the ranges further subdivided throughout the range. Each value within the range and the endpoints of the range are disclosed herein as separate embodiments.

[0060] The detailed description and accompanying drawings support and describe this disclosure, but the scope of this disclosure is defined only by the claims. While some best modes and other embodiments for carrying out the claimed disclosure have been described in detail, various alternative designs and embodiments exist for practicing the disclosure as defined in the appended claims. Furthermore, the features of the embodiments shown in the drawings or the various embodiments mentioned in this specification are not necessarily to be construed as embodiments independent of each other. Rather, each feature described in one of the examples of embodiments may be combined with one or more other desired features from other embodiments to obtain other embodiments not described in words or by reference to the drawings. Therefore, these other embodiments fall within the framework of the appended claims.

Claims

1. A sensing system suitable for receiving visual data from a camera, the sensing system comprising: A controller having a processor and tangible, non-transitory memory on which instructions are recorded; The sub-sampling module, object detection module, and attention module can each be selectively executed by the controller; Wherein, the processor's execution of the instructions causes the controller to: The input image is sampled from the visual data via the sub-sampling module to generate a rescaled whole image frame; Feature data is extracted from the entire rescaled image frame via the object detection module; The region of interest in the entire rescaled image frame is identified, at least in part, based on the output of the attention module; and A first image is generated based on the rescaled entire image frame, and a second image is generated based on the region of interest, the second image having a higher resolution than the first image. The region of interest defines a center, and the controller is adapted to select the center from the region of potential centers based on the output of the attention module.

2. The sensing system according to claim 1, wherein: The camera is mounted to the vehicle; as well as The controller is configured to control the operation of the vehicle in part based on the second image.

3. The sensing system according to claim 1, wherein: The object detection module includes a first backbone unit and a second backbone unit; and The first backbone unit and the second backbone unit are adapted to extract feature data from the input image and the region of interest, respectively.

4. The sensing system according to claim 1, wherein: The controller is adapted to generate a second image based on multiple episode frames over time, each of the multiple episode frames combining feature data from the preceding episode frame.

5. The sensing system according to claim 1, wherein: The output of the attention module includes an attention agent indicating the probability of a positive reward corresponding to each potential center within the region of the potential center.

6. The sensing system according to claim 5, wherein: The output of the attention module includes an attention agent indicating the corresponding predicted reward for each potential center within the region of the potential center.

7. The sensing system according to claim 1, wherein: The attention module includes a deep Q-value network trained to identify regions of interest, and training the deep Q-value network includes maximizing the reward.

8. The sensing system according to claim 7, wherein: Rewards are awarded by comparing a first number of true positives obtained from the basic fact dataset with a second number of true positives obtained from the original dataset.

9. The sensing system according to claim 8, wherein: Training the deep Q-value network includes: obtaining a corresponding loss as the difference between the reward and the predicted reward corresponding to each potential center in the region of the potential center; and minimizing the corresponding loss.

10. A method of operating a sensing system, the sensing system having a camera for collecting visual data and a controller having a processor and tangible non-transitory memory, the method comprising: Visual data is transmitted from the camera to the controller, which is adapted to selectively execute the sub-sampling module, the object detection module, and the attention module; The input image is sampled from the visual data via the sub-sampling module to generate a rescaled full image frame; Feature data is extracted from the entire rescaled image frame via the object detection module; The region of interest in the entire rescaled image frame is identified based on the output of the attention module, wherein the region of interest is defined by a center; The controller selects the center from the region of potential centers based on the output of the attention module; as well as A first image is generated based on the rescaled entire image frame, and a second image is generated based on the region of interest, the second image having a higher resolution than the first image.

11. The method of claim 10, further comprising: Attach the camera to the vehicle; as well as The operation of the vehicle is controlled in part based on the second image via the controller.

12. The method of claim 10, further comprising: The controller is adapted to generate the second image based on a plurality of episode frames over time, each of the plurality of episode frames incorporating feature data from the preceding episode frame.

13. The method of claim 10, further comprising: A deep Q-value network is trained in the attention module to identify the region of interest, and training the deep Q-value network includes maximizing the reward.

14. The method of claim 13, further comprising: Rewards are awarded by comparing a first number of true positives obtained from the basic fact dataset with a second number of true positives obtained from the original dataset.

15. The method of claim 10, further comprising: An attention agent is generated via the attention module, which indicates the probability of a positive reward corresponding to each potential center within the region of the potential center.

16. The method of claim 10, further comprising: An attention agent is generated via the attention module, which indicates the corresponding predicted reward for each potential center within the region of the potential centers.

17. A vehicle comprising: A camera, which is suitable for acquiring visual data; A controller adapted to receive the visual data from the camera, the controller having a processor and tangible non-transitory memory on which instructions are recorded; The sub-sampling module, object detection module, and attention module can each be selectively executed by the controller; Wherein, the processor's execution of the instructions causes the controller to: The input image is sampled from the visual data via the sub-sampling module to generate a rescaled whole image frame; Feature data is extracted from the entire rescaled image frame via the object detection module; The region of interest in the entire rescaled image frame is identified based on the output of the attention module; Generate a second image of the region of interest, the second image having a higher resolution than the input image; and The operation of the vehicle is controlled in part based on the second image. The region of interest defines a center, and the controller is adapted to select the center from the region of potential centers based on the output of the attention module.

Images