Prediction of pedestrian step down (PSD) event which is not regulated

Abstract

To provide a method for a pedestrian step down (PSD) alert which is not regulated.SOLUTION: A method is configured so that, a computerized system of a vehicle performs the steps of: receiving a PSD index which is not regulated and indicates a PSD situation without regulation; acquiring information which is sensed related to an environment of the vehicle; processing the sensed information, the processing searching at least one PSD index which is not regulated; and when at least one PSD identifier which is not regulated is detected, autonomously determining by the vehicle, a fact that a situation approximates to a situation in which step down of a pedestrian to a space where the vehicle can travel in the vehicle environment is predicted. Autonomous determination causes a response with respect to the detection, the response is an instant response, and is executed by the at least computerized system of the vehicle.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] Pedestrians are very vulnerable road users, but they put themselves at risk because they behave in ways that are not in line with traffic rules. One of them is stepping down onto the road when the sidewalk is blocked by various factors, such as parked vehicles, garbage bins, road sweepers cleaning the sidewalk, unloaded luggage, etc. While driving, drivers are usually not aware of passing close to blocked sidewalks and cannot predict or anticipate in advance the movements of pedestrians who leave the sidewalk and step down onto the road to enter the drivable area. Summary of the Invention

[0002] A method, system, and non-transitory computer-readable medium for predicting unregulated pedestrian step-down (PDD) events. [Brief description of the drawings]

[0003] Embodiments of the present disclosure will be understood and more fully appreciated from the following detailed description taken in conjunction with the drawings, in which: [Figure 1A] A diagram showing an example of the method [Figure 1B] A diagram showing an example of a signature [Figure 1C] A diagram showing an example of a dimension expansion process [Figure 1D] A diagram showing an example of a merge operation. [Figure 1E] A diagram showing an example of a hybrid process [Figure 1F] A diagram showing an example of the method [Figure 1G] A diagram showing an example of the method [Figure 1H] A diagram showing an example of the method [Figure 1I] A diagram showing an example of the method [Figure 1J] A diagram showing an example of the method [Figure 1K] A diagram showing an example of the method [Figure 1L] A diagram showing an example of the method [Figure 1M] A diagram showing an example of the system [Figure 1N] FIG. 1 is a partial pictorial, partial block diagram of an exemplary obstacle detection and mapping system constructed and operative in accordance with embodiments described herein; [Figure 1O] A diagram showing an example of the method [Figure 1P] A diagram showing an example of the method [Figure 1Q] FIG. 1 illustrates a method for object detection. [Diagram 2] A diagram showing an example of the method [Diagram 3] A diagram showing an example of the method [Figure 4] Diagram showing an example of the method in action DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The present specification and / or drawings may refer to images. An image is an example of a media unit. Reference to an image may also apply mutatis mutandis to a media unit. A media unit is an example of sensed information. Reference to a media unit may also apply mutatis mutandis to natural signals such as, but not limited to, naturally generated signals, signals representing human actions, signals representing stock market operations, medical signals, etc. Reference to a media unit may also apply mutatis mutandis to sensed information. Sensed information may be sensed by any type of sensor, such as a visible light camera, a sensor sensing infrared, radar imaging, ultrasound, electro-optical, radiography, LIDAR (light detection and ranging), etc.

[0005] The present specification and / or drawings may refer to a processor. A processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, or a combination of such integrated circuits.

[0006] Any combination of any of the steps of the methods illustrated in this specification and / or in the drawings may be provided.

[0007] Any combination of the subject matter of any of the claims may be provided.

[0008] Any combination of the systems, units, components, processors, and sensors illustrated in the specification and / or drawings may be provided.

[0009] A system, method, and non-transitory computer readable medium for providing a method for unregulated pedestrian step-down (PSD) warning may be provided.

[0010] A regulated PSD event is one in which the PSD is performed in accordance with traffic rules and / or traffic regulations, and / or is identified by dedicated traffic signs, and / or has a predefined (legally defined) shape and / or size. An example of a regulated PSD event is a pedestrian crossing a roadway using a crosswalk (such as a zebra crossing). Crosswalks are identified by explicit road signs and may have a shape and size in accordance with laws and regulations.

[0011] An unregulated PSD event is one in which the PSD is not performed in accordance with traffic rules and / or traffic laws, and / or is not identified by a dedicated traffic sign, and / or does not have a predefined (legally defined) shape and / or size. An example of an unregulated regulated PSD event may include a pedestrian attempting to get around one or more trash cans that are placed on a sidewalk and at least partially and temporarily blocking the sidewalk.

[0012] 2 is an example of method 101. Method 101 may be performed by a computerized system in a vehicle, by computerized systems in multiple vehicles (e.g., coordinating, load balancing, allocating different tasks between different vehicles, etc.), by one or more computerized systems not belonging to a vehicle, etc.

[0013] Method 101 may begin by receiving 110, by a computerized system of the vehicle, an uncontrolled pedestrian step-down (PSD) indicator that indicates an uncontrolled PSD condition. Step 110 may be performed one time or on an ongoing basis. The uncontrolled PSD indicator may be provided, updated, replaced, etc.

[0014] An uncontrolled PSD indicator may be an uncontrolled PSD predictor, in which case it may predict the likelihood or likelihood of an uncontrolled PSD before it occurs, e.g., detecting one or more trash cans blocking a sidewalk and predicting that an uncontrolled PSD will occur near these trash cans.

[0015] Method 101 may also include step 120 of obtaining sensed information about the vehicle's environment. The environment may include one or more sidewalks or other paths that are not part of the road but are proximate to the road. Step 120 may be performed continuously or non-continuously during the driving session.

[0016] Step 120 may be followed by step 130 of processing the sensed information, which may include searching for at least one unregulated PSD indicator.

[0017] Step 130 may be followed by step 140 in which the vehicle autonomously determines, upon finding at least one unrestricted PSD identifier, that it is approaching a situation in which a pedestrian is expected to step down into drivable space within the vehicle's environment. If no unrestricted PSDs are found, step 140 may be skipped.

[0018] The determination may also include determining a probability of occurrence of a PSD event. The probability of occurrence may be determined based on statistics learned during generation of the PSD identifier and / or based on additional information about the PSD event (e.g., timing statistics, such as timing of PSD events detected during generation of the PSD identifier). For example, if a pedestrian performs a PSD during a certain time window (e.g., between 7-9 am - but not between 5-6 am), then reaching the PSD identifier outside of the time window reduces the probability of a PSD event, and the response may be probability sensitive - e.g., a PSD indicator used as a predictor may be ignored if its probability of occurrence is below a threshold - e.g., outside of the time window.

[0019] The autonomously deciding may trigger a response to the discovery, where the response is an immediate response and is executed by at least the vehicle's computerized systems, where immediate means within one second, within one to three seconds, etc.

[0020] According to one embodiment, step 140 is followed by step 150 in which at least the vehicle's computerized systems respond to the discovery.

[0021] The response will include at least one of the following: a) Changes in vehicle progress. b) Transfer of control from the automated driving system to driver-assisted driving. c) Transfer of control from assisted driving to automated driving. d) Generating an alarm (whether or not noticeable to a human being) and automatically stopping the vehicle. e) Distance from anticipated unregulated PSD events. f) Acceleration. g) Sending an alert to another vehicle. h) Sending warnings to pedestrians. i) If the uncontrolled PSD indicator is predictive, the vehicle may verify whether an uncontrolled PSD has occurred (e.g., by using a rear camera or by receiving information from other vehicles that have passed the potential PSD area on the roadway). j) updating one or more potential PSD indicators based on validation; k) Update the probability of occurrence, etc. l) Warning to the driver. m) Activation of certain driving modes, e.g. low speed driving, ESA (Emergency Steering Assist) - automatic steering into drivable areas. n) Activation of active safety features (e.g. braking). o) A request and / or instruction from another unit of the vehicle to perform at least one of steps a) to n). p) operating another unit of the vehicle to perform at least one of steps a) to n).

[0022] At least some of the unregulated PSD indicators are cluster signatures, and environmental signatures can be compared to the cluster signatures to detect unregulated PSD events (or anticipated and / or predicted unregulated PSD events).

[0023] The method 101 may include constantly matching the driving footage to a step-down indicator.

[0024] It should be noted that the unregulated PSD indicators may be location and / or time specific, and a data structure of the location and / or timing of events resulting in an unregulated PSD event may be provided - and accessed by the vehicle computer while driving - and upon reaching the location (or combination of location and arrival time) - a response (e.g., the response of step 140) may then be triggered. For example, an event such as blocking a sidewalk may occur in the run-up to emptying a trash can. This data structure may be used in addition to or instead of step 130.

[0025] It should also be noted that method 100 is applicable to detecting restricted PSD events (eg, reaching a crosswalk) in addition to detecting unrestricted PSD events.

[0026] PSD events can also occur randomly or for no apparent reason. Often these random PSD events cannot be predicted. Also, if a PSD event occurs enough times, it may be possible to detect it.

[0027] FIG. 3 is an example of a method 201 for generating an unregulated PSD index.

[0028] The method 201 may begin by obtaining 210 a first group of untagged videos.

[0029] Step 210 may be followed by step 220 of locating unregulated PSD events within the first group of untagged videos.

[0030] Step 220 may be followed by step 230 of generating uncensored PSD indicia by a generation process that includes processing the prior uncensored PSD event video segments, each of which begins a predefined time period (e.g., 2-20 seconds) before a corresponding uncensored PSD event of the prior uncensored PSD event. This processing includes finding indicia that distinguish the video segments of the prior uncensored PSD event from video segments associated with the PSD event (e.g., video segments captured at a time substantially different from the PSD event).

[0031] Step 230 includes at least some of the following steps: · Removing identifiers of objects that appear in the a priori unregulated PSD event video segments but that are not characteristic of the a priori unregulated PSD event, step 231. Removal may include searching for objects that appear with more than a predefined probability in video segments that are distinct from the a priori unregulated PSD event video segments. · Step 232 of generating a pre-qualified PSD event video segment signature. A step 233 of clustering the pre-regulated PSD event video segment signatures to provide a number of clusters. · Generating cluster signatures for the plurality of clusters, step 234. Classifying pedestrians into multiple pedestrian classes 235. The classes may not be known in advance or may be discovered based on statistics of discovered PSD events. · Step 236 of associating pedestrian classes with unregulated PSD indicators associated with the pedestrian classes. Different classes of pedestrians may be associated with different clusters. Step 230 may determine a correspondence between pedestrian classes and PDS events. · Associating probability of occurrence information with unregulated PSD indicators 237. The number of occurrences of PSD events identified by the unregulated PSD indicators can be mapped (using any probability function) to a probability of occurrence (e.g. the ratio of the number of occurrences of a particular PSD event to the total number of occurrences of detected PSD events).

[0032] The generation of the unregulated PSD indicator may, for example, include at least some of the following steps: Collecting untagged video streams (such as dashcams or other car cameras). · Apply any technique to detect drivable space. · Apply all pedestrian detection techniques. Identify PSD events in which pedestrians enter drivable spaces in any untagged video stream. Whenever a PSD event is captured by any of the untagged videos, extract a video segment that may start during a predefined period (e.g., 1-10 seconds, 5-20 seconds, 2-30 seconds, up to 1 minute, more than 1 minute) before the PSD event and end at the PSD event or end after the PSD event. · Unsupervised clustering of video segments to identify different PSD events or any kind of object or scene that can cause a step-down event (e.g. parked cars on sidewalk, x cm of sidewalk available + pedestrian with stroller, washing machine / trash can on sidewalk, etc.). Examples of objects that can be clustered: specific elements that obstruct pedestrians' path on the sidewalk (e.g. parked cars, road maintenance / construction, garbage trucks, trash cans, vehicle unloading equipment), or scenarios that force certain pedestrians to step down (e.g. limited space on sidewalk and pedestrian with stroller / wheelchair / shopping cart), different environments (rural / urban, weather / lighting conditions). Take a random video and remove clustered objects (such as traffic lights or pavement tiles) that appear with a high probability in the random video. Save as concepts / models the objects that were successfully clustered but not removed in the random set, as there is a possibility that certain combinations of clustered concepts may appear (e.g. stroller + trash can, i.e. people without strollers can get through but not on the road, whereas strollers cannot).

[0033] FIG. 4 illustrates an example of the implementation of the method 101.

[0034] A vehicle 330 is traveling in an environment 300. The environment includes a road including a current lane 301 (the vehicle 303 is traveling in the current lane 301) and an oncoming lane 302. There is also a nearby sidewalk 303 and another sidewalk 304. An onboard camera captures a group of four truck cans installed near the sidewalk 303. A first group of trash cans 311 includes a row of four trash cans aligned parallel to the edge of the sidewalk. A second group of trash cans 312 includes three trash cans forming a triangle. A third group of trash cans 313 includes a pair of trash cans. A fourth group of trash cans 314 is a single trash can. The unregulated PSD indicator indicates which group (if any) is associated with the PSD event. If one of the groups is associated with the unregulated PSD indicator, step 140 may be performed by the vehicle 330. FIG. 4 also illustrates an example data structure 333 for an unregulated PSD event.

[0035] It should be noted that a data structure for unregulated PSD events may be formed, such as a map that includes location and / or timing information related to the occurrence (known and / or expected future occurrences) of unregulated PSD events and / or other metadata (e.g., type of PSD event).

[0036] Future drivers / autonomous vehicles can be alerted according to their location before arriving at the location of an unregulated PSD event. This data structure may be dynamic in its contents and may be updated at any update frequency (e.g., every few minutes, hourly, etc.). Furthermore, the data structure may be analyzed to derive specific rules (e.g., certain locations are usually blocked at certain times / days of the week, etc.).

[0037] Low-power generation of signatures Analysis of the content of a media unit can be performed by generating a signature of the media unit and comparing the signature to reference signatures. The references may be arranged in one or more conceptual structures or in other ways. The signatures can be used for object detection and other applications.

[0038] A signature can be generated by creating a multidimensional representation of a media unit, which may have a large number of dimensions. A high number of dimensions may ensure that the multidimensional representations of different media units containing different objects are sparse, and that the object identifiers of different objects are distant from each other.

[0039] The generation of the signature is performed in an iterative manner involving multiple iterations, where each iteration may involve a merge operation followed by an extend operation. The extend operation of an iteration is performed by the spanning elements of that iteration. At each iteration, a significant amount of power can be saved by determining which spanning elements (of that iteration) are relevant and reducing the power consumption of the non-relevant spanning elements.

[0040] In many cases, most of the spanning elements of an iteration are irrelevant, so after determining their relevance (by the spanning elements), the spanning elements deemed irrelevant are shut down or put into an idle mode.

[0041] FIG. 1A illustrates a method 5000 for generating a signature of a media unit.

[0042] The method 5000 may begin by step 5010 of receiving or generating sensed information.

[0043] The sensed information may be media units of multiple objects.

[0044] Step 5010 may continue by processing the media unit by performing multiple iterations, at least some of the multiple iterations including applying a dimensional expansion operation followed by a merging operation, depending on the spanning element of the iteration.

[0045] Processing may include: Step 5020 of performing the kth iteration expansion process (k may be a variable used to track the number of iterations); Step 5030 of performing the k-th iteration merging process; Step 5040 of changing the value of k; Step 5050 to check if all necessary iterations have been performed; if so, proceed to step 5060 to complete the signature generation; otherwise, jump to step 5020.

[0046] The output of step 5020 is the kth iterative expansion result 5120.

[0047] The output of step 5030 is the kth iteration merge result 5130.

[0048] Each iteration (except the first one) - the merged result of the previous iteration is the input of the current iteration expansion step.

[0049] At least some of the K iterations involve selectively reducing the power consumption of some spanning elements that are deemed irrelevant (during step 5020).

[0050] FIG. 1B is an example of an image signature 6027 for a media unit that is an image 6000 and the result 6013 of the last (Kth) iteration.

[0051] An image 6001 is virtually divided into segments 6000(i,k), which may, but do not necessarily, have the same shape and size.

[0052] The outcome 6013 may be a tensor that contains a vector of values ​​for each segment of a media unit. One or more objects may appear in a segment. For each object, an object identifier (in the signature) points to the location of a significant value in a particular vector associated with a particular segment.

[0053] For example, the top left segment of the image (6001(1,1)) may be represented in the result 6013 by the multi-valued vector V(1,1) 6017(1,1). The number of values ​​per vector may exceed 100, 200, 500, 1000, etc.

[0054] Significant values ​​can be selected (e.g., 10 or more values, 20 or more values, 30 or more values, 40 or more values, and / or 0.1%, 0.2%, 0.5%, 1%, 5% of all values ​​in the vector, etc.). Significant values ​​can have a value -, but may be selected in other ways.

[0055] 1B illustrates a set of significant responses 6015(1,1) for vector V(1,1) 6017(1,1), including five significance values, such as a first significance value SV1(1,1) 6013(1,1,1), a second significance value SV2(1,1), a third significance value SV3(1,1), a fourth significance value SV4(1,1), and a fifth significance value SV5(1,1) 6013(1,1,5).

[0056] The image signature 6027 includes five indexes for searching five significant values. The first to fifth identifiers ID1 to ID5 are indexes for searching the first to fifth significant values.

[0057] FIG. 1C shows the k-th iterative expansion step.

[0058] The kth iteration extension process begins by receiving the merged result 5060' of the previous iteration.

[0059] The merged result of a previous iteration may include a value indicative of a previous dilation step - for example, a value indicative of an associated spanning element from a previous dilation operation, a value indicative of an associated region of interest in a multidimensional representation of the merged result of a previous iteration.

[0060] The merged results (from the previous iteration) are provided to spanning elements, such as spanning elements 5061(1) through 5061(J).

[0061] Each spanning element is associated with a unique set of values. The set may contain one or more values. The spanning elements apply different functions that may be orthogonal to each other. The use of non-orthogonal functions may increase the number of spanning elements, but this increase may be tolerable.

[0062] The spanning elements can have decorative functions applied to them, even though they are not orthogonal to each other.

[0063] A spanning element may be associated with different combinations of object identifiers that "cover" multiple possible media units. Candidate combinations of object identifiers may be selected in various ways, e.g., based on their appearance in various images (e.g., test images), randomly, pseudo-randomly, according to some rule, etc. Of these candidates, combinations may be selected to cover the multiple possible media units and / or to be decoratively related such that certain objects are mapped to the same spanning element.

[0064] Each spanning element compares the merged result value with the unique set (associated with the spanning element) and if there is a match, the spanning element is considered relevant. If so, the spanning element completes the extend operation.

[0065] If there is no match, the spanning element is considered irrelevant and enters a low power mode, which is also called idle mode, standby mode, etc. The low power mode is called low power because the power consumption of irrelevant spanning elements is lower than the power consumption of associated spanning elements.

[0066] In FIG. 1C, various spanning elements are related (5061(1)-5061(3)) and one spanning element is unrelated (5061(J)).

[0067] Each associated spanning element may perform a spanning operation that includes assigning an output value that indicates the identity of the associated spanning element of the iteration. The output value may also indicate the identity of a previous associated spanning element (from a previous iteration).

[0068] For example - assume element number 50 is relevant and associated with a set of unique values ​​of 8 and 4 - the output value could reflect the numbers 50, 4 and 8 - say 1000 multiplied by (50+40)+40. Other mapping functions can also be applied.

[0069] FIG. 1C also shows the steps performed by each spanning element:

[0070] It is checked whether the merge result is related to a spanning element (step 5091).

[0071] If so, the spanning operation is complete (step 5093).

[0072] If not, an idle state is entered (step 5092).

[0073] FIG. 1D shows examples of various merge operations.

[0074] The merging operation may include finding regions of interest, which are regions within the multi-dimensional representation of the sensed information, that may exhibit a more significant response (e.g., a stronger, higher intensity response).

[0075] The merge operation (performed during the kth iteration merge operation) may include at least one of the following:

[0076] Step 5031 of searching for overlaps between the regions of interest (resulting from the k-th iterative dilation operation) and defining a region of interest associated with the overlap.

[0077] Step 5032 determines whether to drop one or more regions of interest and does so according to the determination.

[0078] Step 5033 searches for relationships between the regions of interest (results of the k-th iterative expansion operation) and defines the regions of interest that are in a relationship.

[0079] Search for nearby regions of interest (in the k-th iterative expansion operation result) and define the regions of interest associated with proximity, step 5034. Proximity may be a distance that is a percentage (e.g., less than 1%) of the multidimensional space, or may be a percentage of at least one of the regions of interest that are checked for proximity.

[0080] Step 5035 searches for relationships between the regions of interest (results of the k-th iterative expansion operation) and defines the regions of interest that are in the relationships.

[0081] A step 5036 of merging and / or dropping the kth iteration region of interest based on shape information associated with the shape of the kth iteration region of interest.

[0082] The same merge operation may be applied in different iterations.

[0083] Alternatively, different merge operations may be performed during different iterations.

[0084] FIG. 1E shows an example of hybrid processing and an input image 6001.

[0085] The hybrid process is hybrid in the sense that some of the expand and merge operations are performed by a convolutional neural network (CNN) and some of the expand and merge operations (representing additional iterations of expand and merge) are not performed by the CNN but by a process that determines the relevance of spanning elements and may include putting irrelevant spanning elements into a low power mode.

[0086] In FIG. 1E, one or more initial iterations are performed by first and second CNN layers 6010(1) and 6010(2) applying first and second functions 6015(1) and 6015(2).

[0087] The output of these layers provided information about image properties that do not necessarily result in object detection. Image properties include edge locations, curve properties, etc.

[0088] The CNN may include additional layers (e.g., third through Nth layers 6010(N)) that may provide a CNN output 6018 that may include object detection information. However, additional layers may not be included.

[0089] It should be noted that performing the entire signature generation process by a hardware CNN with fixed connectivity may result in high power consumption, since the CNN cannot reduce the power consumption of irrelevant nodes.

[0090] FIG. 1F illustrates a method 7000 for calculating a signature with low power consumption.

[0091] The method 7000 begins by step 7010 of receiving or generating media units for a plurality of objects.

[0092] Step 7010 may be followed by step 7012 of processing the media unit by performing multiple iterations, where at least some of the multiple iterations include applying a dimensional expansion process followed by a merging operation, depending on the spanning element of the iteration.

[0093] Applying the iterative dimensionality expansion process may include (a) determining the relevance of the iterative spanning elements; and (b) completing the dimensionality expansion process with the iterative relevant spanning elements and reducing power consumption of irrelevant spanning elements at least until completion of application of the dimensionality expansion process.

[0094] The identifier may be search information for retrieving the portion of interest.

[0095] At least some of the plurality of iterations may be a majority of the plurality of iterations.

[0096] The output of the multiple iterations may include multiple characteristic attributes for each segment of the multiple segments of the media unit, and a significant portion of the output of the multiple iterations may include the more impactful characteristic attributes.

[0097] A first iteration of the multiple iterations may include applying a dimension expansion process by applying a different filter to the media unit.

[0098] At least some of the multiple repeats exclude at least the first repeat of the multiple repeats, see, e.g., FIG. 1E.

[0099] The determination of the relatedness of spanning elements of a repeat may be based on the identity of at least some of the related spanning elements of at least one previous repeat.

[0100] Determining the relatedness of spanning elements of a repeat may be based on the identity of at least some of the related spanning elements of at least one previous repeat that precedes the repeat.

[0101] Determining the relevance of a repeat spanning element may be based on characteristics of the media unit.

[0102] The determination of the relevance of a spanning element of an iteration may be performed by the spanning element of the iteration.

[0103] The method 7000 may be performed by one or more layers of a neural network and may include neural network processing operations that do not belong to at least some of the multiple iterations. See, for example, FIG. 1E.

[0104] At least one iteration may be performed without reducing the power consumption of irrelevant neurons in one or more layers.

[0105] One or more layers may output information about characteristics of a media unit, which information differs from the recognition of multiple objects.

[0106] Applying a dimensionality expansion process with spanning elements of an iteration different from the first iteration may include assigning an output value indicative of the identity of the associated spanning element of the iteration, see, e.g., FIG. 1C.

[0107] Applying the dimensional expansion process across elements of an iteration different from the initial iteration may include assigning an output value indicative of the history of the dimensional expansion process up to the iteration different from the initial iteration.

[0108] Each spanning element is associated with a subset of the reference identifiers. The determination of the relevance of each spanning element of an iteration may be based on a relationship between the subset of reference identifiers of the spanning element and the output of the last merge operation before the iteration.

[0109] The output of the iterative dimensional expansion steps may be a multi-dimensional representation of the media unit, which may include a media unit region of interest that may be associated with one or more of the expansion steps that produced the region of interest.

[0110] The iterative merging operation may include selecting subgroups of the media unit regions of interest based on spatial relationships between the subgroups of the multi-dimensional regions of interest.

[0111] The method 7000 may include applying a merge function to subgroups of the multidimensional region of interest. See, e.g., FIG. 1C.

[0112] The method 7000 may include applying an intersection function to a subgroup of the multi-dimensional region of interest. See, e.g., FIG. 1C.

[0113] The merging operation of the iterations may be based on the actual size of one or more multi-dimensional regions of interest.

[0114] The iterative merging operation may be based on the relationship between the sizes of the multi-dimensional regions of interest, for example, larger multi-dimensional regions of interest may be kept and smaller multi-dimensional regions of interest may be ignored.

[0115] The merging operation of a repetition may be based on at least a change in the region of interest of a media unit between the repetition and one or more previous repetitions.

[0116] Step 7012 may be followed by step 7014 of determining an identifier associated with a significant portion of the output of the multiple iterations.

[0117] Step 7014 may be followed by step 7016 of providing a signature that includes the identifier and represents the plurality of objects.

[0118] Localization and Segmentation All of the above mentioned signature generation methods provide signatures that do not explicitly contain precise shape information, which makes the signatures more robust to shape-related inaccuracies and other shape-related parameters.

[0119] The signature includes an identifier for identifying an area of ​​interest in the media.

[0120] Each media region of interest may represent an object (e.g., a vehicle, a pedestrian, a road element, a human-made structure, a wearable, a shoe, a natural element such as a tree, sky, sun, etc.) or a part of an object (e.g., for a pedestrian, the neck, head, arm, leg, thigh, hip, foot, upper arm, forearm, wrist, hand). It should be noted that for the purposes of object detection, a part of an object may be considered an object.

[0121] The exact shape of the object may be of interest.

[0122] FIG. 1G illustrates a method 7002 for generating a hybrid representation of a media unit.

[0123] The method 7002 may include a sequence of steps 7020, 7022, 7024 and 7026.

[0124] Step 7020 may include receiving or generating a media unit.

[0125] Step 7022 may include processing the media unit by performing multiple iterations, at least some of the multiple iterations including applying a dimensional expansion operation followed by a merging operation, depending on the spanning element of the iteration.

[0126] Step 7024 may include selecting, based on the output of the multiple iterations, a media unit region of interest that contributed to the output of the multiple iterations.

[0127] Step 7026 may include providing a hybrid representation, where the hybrid representation may include (a) shape information regarding a shape of the region of interest of the media unit, and (b) a media unit signature including an identifier that identifies the region of interest of the media unit.

[0128] Step 7024 may include selecting a media region of interest for each segment from a plurality of segments of the media unit, see, for example, FIG.

[0129] Step 7026 may include a step 7027 of generating shape information.

[0130] The shape information may include polygons that represent shapes that substantially bound the region of interest of the media unit. These polygons may be of high degree.

[0131] To conserve storage space, the method may include a step 7028 of compressing the geometry information of the media unit to provide compressed geometry information of the media unit.

[0132] FIG. 1H illustrates a method 5002 for generating a hybrid representation of a media unit.

[0133] The method 5002 may begin by step 5011 of receiving or generating a media unit.

[0134] Step 5011 may continue by processing the media unit by performing multiple iterations, at least some of the multiple iterations including applying a dimensional expansion operation followed by a merge operation, depending on the spanning element of the iteration.

[0135] The process continues with steps 5060 and 5062 .

[0136] The process may include steps 5020, 5030, 5040, and 5050.

[0137] Step 5020 may include performing the kth iterative expansion process (k may be a variable used to track the number of iterations).

[0138] Step 5030 may include performing the k-th iterative merging process.

[0139] Step 5040 involves changing the value of k.

[0140] Step 5050 involves checking if all necessary iterations have been performed, and if so, proceeding to steps 5060 and 5062. If not, jumping to step 5020.

[0141] The output of step 5020 is the kth iterative expansion result.

[0142] The output of step 5030 is the kth iteration merging result.

[0143] Each iteration (except the first one) - the merged result of the previous iteration is the input of the current iteration expansion step.

[0144] Step 5060 may include completing the generation of the signature.

[0145] Step 5062 may include generating shape information regarding the shape of the region of interest of the media unit. The signature and the shape information provide a hybrid representation of the media unit.

[0146] The combination of steps 5060 and 5062 corresponds to providing a hybrid representation, which may include (a) shape information regarding the shape of the media unit region of interest, and (b) a media unit signature including an identifier that identifies the media unit region of interest.

[0147] Object detection using compressed shape information Object detection may involve comparing a signature of an input image with signatures of one or more cluster structures to find one or more cluster structures that contain one or more matching signatures that match the signature of the input image.

[0148] The number of input images compared to the cluster structures can far exceed the number of cluster structure signatures. For example, thousands, tens of thousands, hundreds of thousands (or even more) of input signatures can be compared to signatures of a much smaller number of cluster structures. The ratio of the number of input images to the total number of signatures of all cluster structures can exceed 10, 100, 1000, etc.

[0149] To save computational resources, the shape information of the input image is sometimes compressed.

[0150] On the other hand, the shape information of the signatures belonging to the cluster structure is uncompressed and may be more accurate than compressed shape information.

[0151] If higher quality is not required, the shape information of the cluster signatures can also be compressed.

[0152] Compression of the shape information of a cluster signature may be based on the priority of the cluster signature, the popularity of matching with the cluster signature, and the like.

[0153] Shape information associated with the input images that match one or more of the cluster structures may be calculated based on the shape information associated with the matching signatures.

[0154] For example, shape information for a particular identifier in a signature of an input image may be determined based on shape information associated with a particular identifier in a matching signature.

[0155] Any operation on the shape information associated with a particular identifier in the matching signature can be applied to determine (more precise) shape information of the region of interest in the input image identified by the particular identifier.

[0156] For example, shapes can be virtually overlaid and the shape defined by a pixel-by-pixel population.

[0157] For example, only pixels that appear in at least half of the overlaid shapes should be considered to belong to the region of interest.

[0158] Other operations include smoothing the overlapped shapes and selecting pixels that appear in all overlapped shapes.

[0159] The compressed shape information may be ignored or may be taken into account.

[0160] FIG. 1I is a diagram illustrating the matching process and the generation of more accurate shape information.

[0161] There are assumed to be multiple (M) cluster structures 4974(1)-4974(M), each of which includes a cluster signature, metadata about the cluster signature, and shape information about a region of interest identified by an identifier in the cluster signature.

[0162] For example, a first cluster structure 4974(1) includes multiple (N1) signatures (referred to as cluster signatures CS) CS(1,1) through CS(1,N1) 4975(1,1) through 4975(1,N1), metadata 4976(1), and shape information (Shapeinfo 4977(1)) regarding the shape of a region of interest associated with an identifier of the CS.

[0163] As yet another example, the Mth cluster structure 4974(M) includes multiple (N2) signatures (called cluster signatures CS) CS(M,1) to CS(M,N2) 4975(M,1) to 4975(M,N2), metadata 4976(M), and shape information (Shapeinfo 4977(M)) regarding the shape of the region of interest associated with an identifier of the CS.

[0164] For example, a cluster reduction attempt may remove a CS from the structure to provide a reduced cluster structure, and the reduced structure is checked to determine whether the reduced cluster signature may still identify an object that was associated with the (non-reduced) cluster signature, and if so, the signature is reduced from the cluster signature.

[0165] The signatures of each cluster structure are associated with one another based on the similarity of the signatures and / or based on associations between metadata of the signatures.

[0166] Assuming that each cluster structure is associated with a unique object, the object of a media unit can be identified by finding the cluster structure associated with that object. Finding a matching cluster structure may include comparing a signature of the media unit to the signatures of the cluster structures and searching for one or more matching signatures among the cluster signatures.

[0167] FIG. 1I - A media unit with a hybrid representation undergoes object detection. The hybrid representation includes a media unit signature 4972 and compressed shape information 4973.

[0168] The media unit signature 4972 is compared to the signatures of M cluster structures CS(1,1) 4975(1,1) through CS(M,N2) 4975(M,N2).

[0169] Assume that one or more cluster structures are matching cluster structures.

[0170] If a matching cluster structure is found, the process proceeds by generating shape information that is more accurate than the compressed shape information.

[0171] The shape information is generated for each identifier.

[0172] For each j in the range 1 to J, where J is the number of identifiers per media unit signature 4972, the method may perform the following steps:

[0173] Shape information for the jth identifier of each matching signature, or shape information for each signature of the matching cluster structure, is determined (step 4978(j)).

[0174] Shape information for the j-th identifier is generated with higher accuracy (step 4979(j)).

[0175] For example, if the matching signatures include CS(1,1) 2975(1,1), CS(2,5) 2975(2,5), CS(7,3) 2975(7,3), and CS(15,2) 2975(15,2), and the jth identifier is included in CS(1,1) 2975(1,1), CS(7,3) 2975(7,3), and CS(15,2) 2975(15,2), then the shape information of the jth identifier of the media unit is determined based on the shape information associated with CS(1,1) 2975(1,1), CS(7,3) 2975(7,3), and CS(15,2) 2975(15,2).

[0176] 1P shows an image 8000 that includes four regions of interest 8001, 8002, 8003, 8004. A signature 8010 of the image 8000 includes various identifiers including ID1 8011, ID2 8012, ID3 8013, ID4 8014 that identify the four regions of interest 8001, 8002, 8003, 8004.

[0177] The shapes of the three regions of interest 8001, 8002, 8003, 8004 are four-sided polygons. Precise shape information regarding the shapes of these regions of interest may be generated during the generation of the signature 8010.

[0178] FIG. 1J illustrates a method 8030 for detecting objects.

[0179] Method 8030 may include the steps of method 8020 or may precede steps 8022, 8024, and 8026.

[0180] The method 8030 may include a sequence of steps 8032, 8034, 8036 and 8038.

[0181] Step 8032 may include receiving or generating an input image.

[0182] Step 8034 may include generating a signature of the input image.

[0183] Step 8036 may include comparing the signature of the input image to the signature of a particular conceptual structure. A conceptual structure is generated by the method 8020.

[0184] Step 8038 may include determining that the input image contains an object if at least one of the signatures of the particular conceptual structure matches a signature of the input image.

[0185] FIG. 2D illustrates a method 8040 for object detection.

[0186] Method 8040 may include the steps of method 8020 and may be preceded by steps 8022, 8024, and 8026.

[0187] The method 8040 may include a sequence of steps 8041, 8043, 8045, 8047 and 8049.

[0188] Step 8041 involves receiving or generating an input image.

[0189] Step 8043 may include generating a signature of the input image, where the signature of the input image includes only a portion of the particular second image identifier.

[0190] Step 8045 may include rescaling the input image to a first scale to provide a corrected input image.

[0191] Step 8047 involves generating a signature of the modified input image.

[0192] Step 8049 may include verifying that the input image contains an object if the signature of the corrected input image contains at least one particular first image identifier.

[0193] Robust object detection for various shooting angles Object detection may advantageously be robust to the acquisition angle (the angle between the optical axis of the image sensor and some part of the object), which may make the detection process more reliable and use fewer clusters (multiple clusters may not be needed to identify the same object from different images).

[0194] FIG. 1K shows a method 8120 that includes the following steps: Step 8122 receives or generates images of the object taken from different angles. Images of the object taken from different angles that are close to each other are found, step 8124. Close enough may be 1 degree, 5 degrees, 10 degrees, 15 degrees, 20 degrees or less, but the closeness may be better reflected by receipt of substantially the same signature. Linking images with similar signatures, step 8126. This includes looking for local similarities. The similarity is local in the sense that it is calculated for each subset of signatures. For example - assuming similarity is determined for every two images - a first signature can be linked to a second signature that is similar to the first image. A third signature is linked to a second image based on the similarity of the second and third signatures and regardless of the relationship of the first and third signatures.

[0195] Step 8126 may include generating a concept data structure that includes the similarity signatures.

[0196] This so-called local or sliding window approach, in addition to acquiring enough images (covering a statistically large angle), can generate a conceptual structure that contains the signature of an object photographed from multiple directions.

[0197] Signature Tailored Matching Threshold Object detection can be performed by (a) receiving or generating a conceptual structure that includes signatures of media units and associated metadata, (b) receiving a new media unit and generating a new media unit signature, and (c) comparing the new media unit signature to the concept signatures in the conceptual structure.

[0198] The comparison may include comparing a signature identifier of the new media unit (an identifier of an object that appears in the new media unit) with the concept signature identifier and determining whether the signature of the new media unit matches the concept signature based on signature matching criteria. If such a match is found, the new media unit is considered to contain an object associated with the conceptual structure.

[0199] It has been found that by applying adjustable signature matching criteria, the matching process can be very effective and can adapt to the statistics of identifier occurrence in different scenarios. For example, if a relatively late but very distinct identifier appears in the new media unit signature and cluster signature, a match is obtained, whereas if multiple common but slightly distinct identifiers appear in the new media unit signature and cluster signature, a mismatch may be declared.

[0200] FIG. 1L illustrates a method 8200 for object detection.

[0201] The method 8200 may include:

[0202] Step 8210 receives an input image.

[0203] Step 8212 generates a signature of the input image.

[0204] A step 8214 of comparing the signature of the input image with the signature of the conceptual structure.

[0205] Step 8216 determines whether the signature of the input image matches any of the signatures of the conceptual structure based on signature matching criteria, where each signature of the conceptual structure is associated within a signature matching criterion determined based on the object detection parameters of the signature.

[0206] Step 8218 concludes that the input image contains an object related to the conceptual structure based on the determination result.

[0207] The matching criterion for a signature can be the minimum number of identifiers that show a match. For example, given a signature that contains dozens of identifiers, the minimum number can vary between a single identifier to all identifiers in the signature.

[0208] It should be noted that an input image may contain multiple objects, and the signature of the input image may match multiple cluster structures. The method 8200 is applicable to all matching processes, and the signature matching criteria can be set for each signature of each cluster structure.

[0209] Step 8210 may precede step 8202 of determining each signature matching criterion by evaluating the object detection ability of the signatures under different signature matching criteria.

[0210] Step 8202 may include:

[0211] A step 8203 of receiving or generating a signature for a set of test images.

[0212] For each of the different signature matching criteria, calculate 8204 the object detection capability of the signature.

[0213] Step 8206 of selecting a signature matching criterion based on the object detection ability of the signature under different signature matching criteria.

[0214] Object detection performance can reflect the percentage of signatures in a set of test images that match the signature.

[0215] Selecting the signature matching criterion includes selecting a signature matching criterion that, once applied, causes the percentage of signatures in the test image set that match the signature to close to a predefined desired percentage of signatures in the test image set that match the signature.

[0216] Object detection performance may reflect significant changes in the percentage of a signature in a set of test images that match the signature. For example, assume that the signature matching criterion is a minimum number of identifier matches, and changing the value of the minimum number of matches changes the match rate of test images. A significant change in percentage (e.g., a change of 10%, 20%, 30%, 40% or more) may indicate a desired value. The desired value may be set before a substantial change, near a substantial change, etc.

[0217] For example, referring to Figure 1I, cluster signatures CS(1,1), CS(2,5), CS(7,3), and CS(15,2) match unit signature 4972. Each of these matches can have its own signature matching criteria applied.

[0218] Example System FIG. 1M illustrates an example of a system capable of performing one or more of the methods described above.

[0219] A system includes various components, elements, and units.

[0220] The components and / or units may be processing circuitry implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, or a combination of such integrated circuits.

[0221] Alternatively, each component and / or unit may be implemented in hardware, firmware, or software that can be executed by a processing circuit.

[0222] The system 4900 may include a sensing unit 4902, a communication unit 4904, an input 4911, one or more processors, such as processor 4950, and an output 4919. The communication unit 4904 may include inputs and / or outputs. The communication unit 4904 may communicate with any entity, such as within the vehicle (e.g., driver devices, passenger devices, multimedia devices, etc.), outside the vehicle (other vehicles, other computerized systems (such as the off-vehicle computerized system 4820 in FIG. 1N), other road users, other humans outside the vehicle), etc.

[0223] The inputs and / or outputs may be any suitable communication components, such as network interface cards, universal serial bus (USB) ports, disk readers, modems, transceivers, etc., operable to communicate directly or indirectly with other elements of the system using protocols as are known in the art.

[0224] The processor 4950 may include at least some of the out-of-outs (and thus may not include at least one out-of-out): · Multiple spanning elements 4951(q). · Multiple merge elements 4952(r). · Object Detector 4953. Cluster Manager 4954. -Controller 4955. ·Selection Unit 4956. Object detection determination unit 4957. Signature Generator 4958. ·Mobile Information Unit 4959. -Identifier unit 4960.

[0225] It should be noted that while the system 4900 includes a sensing unit 4902, the sensing unit may receive sensed information from other sensors and / or the sensing unit does not belong to the system. The system can receive information from one or more sensors installed on, associated with, and / or external to the vehicle.

[0226] Any of the methods illustrated herein may be performed fully or partially by system 4900, and / or may be performed fully or partially by one or more other computerized systems, and / or may be performed by one or more computerized systems - for example, by task allocation between computerized systems, by cooperation between multiple computerized systems (e.g., exchange of information, exchange of decisions, optional allocation of resources, joint decision-making, etc.).

[0227] The one or more other computerized systems may be, for example, off-vehicle computerized system 4820 of FIG. 1N, other off-vehicle computerized systems, one or more other in-vehicle systems, a computerized device of a person in the vehicle, or a computerized system off-vehicle (e.g., including a computerized system in another vehicle).

[0228] An example of another in-vehicle system is shown in FIG. 1N as 4830 and is located within a vehicle 4800 traveling along a road 4820 .

[0229] The system 4900 can obtain sensed information from any type of sensor, such as a camera, one or more sensors implemented using any suitable imaging technology instead of or in addition to a conventional camera, infrared sensors, radar, ultrasonic sensors, any electro-optical sensors, radiation sensors, LIDAR (light detection and ranging), telemetry ECU sensors, impact sensors, etc.

[0230] System 4900 and / or other in-vehicle systems may use supervised and / or unsupervised learning to perform any of the methods performed by them, as indicated at 4830.

[0231] The other in-vehicle system 4830 may be an autonomous driving system, an advanced driving assistance system, or a system different from either an autonomous driving system or an advanced driving assistance system.

[0232] The other in-vehicle system 4830 may include the processing circuit 210, the input / output (I / O) module 220, one or more sensors 233, and the database 270. The processing circuit 210 may perform tasks that it is assigned or programmed to perform in connection with any of the methods illustrated herein. Alternatively, the other in-vehicle system 4830 may include another module for performing such tasks (alone or in conjunction with the processing circuit). For example, the processing circuit may execute instructions to provide an autonomous driving manager function. Alternatively, another circuit or module of the in-vehicle system 4830 may provide the autonomous driving manager function.

[0233] FIG. 10 illustrates a method 7002 for generating a hybrid representation of a media unit.

[0234] The method 7002 may include a sequence of steps 7020, 7022, 7024 and 7026.

[0235] Step 7020 may include receiving or generating a media unit.

[0236] Step 7022 may include processing the media unit by performing multiple iterations, at least some of the multiple iterations including applying a dimensional expansion operation followed by a merging operation, depending on the spanning element of the iteration.

[0237] Step 7024 may include selecting, based on the output of the multiple iterations, a media unit region of interest that contributed to the output of the multiple iterations.

[0238] Step 7026 may include providing a hybrid representation, where the hybrid representation may include (a) shape information regarding a shape of the region of interest of the media unit, and (b) a media unit signature including an identifier that identifies the region of interest of the media unit.

[0239] Step 7024 may include selecting a media region of interest for each segment from a plurality of segments of the media unit, see, for example, FIG.

[0240] Step 7026 may include a step 7027 of generating shape information.

[0241] The shape information may include polygons that represent shapes that substantially bound the region of interest of the media unit. These polygons may be of high degree.

[0242] To conserve storage space, the method may include a step 7028 of compressing the geometry information of the media unit to provide compressed geometry information of the media unit.

[0243] FIG. 1P illustrates a method 8020 for scale-invariant object detection.

[0244] The method 8020 may include a first sequence of steps, which may include steps 8022, 8024, 8026 and 8028.

[0245] Step 8022 may include receiving or generating a first image in which the object is displayed at a first scale and a second image in which the object is displayed at a second scale different from the first scale.

[0246] Step 8024 includes generating a first image signature and a second image signature.

[0247] The first image signature includes a first group of at least one particular first image identifier that identifies at least a portion of the object.

[0248] The second image signature includes a second group of specific second image identifiers that identify different portions of the object.

[0249] The second group is larger than the first group.

[0250] Step 8026 may include linking between at least one particular first image identifier and a particular second image identifier.

[0251] Step 8026 may include a link between the first image signature, the second image signature, and the object.

[0252] Step 8026 may include adding the first signature and the second signature to a particular conceptual structure associated with the object.

[0253] Step 8028 may include determining, based at least in part on the link, whether the input image includes an object that is different from the first and second images.

[0254] Determining may include determining that the input image includes the object if a signature of the input image includes at least one of the particular first image identifier or the particular second image identifier.

[0255] The determining may include determining that the input image includes the object if the signature of the input image includes only a portion of at least one particular first image identifier or includes only a portion of a particular second image identifier.

[0256] Linking can be performed for two or more images in which an object is shown at two or more scales.

[0257] FIG. 1Q illustrates a method 8200 for object detection.

[0258] The method 8200 may include:

[0259] Step 8210 receives an input image.

[0260] Step 8212 generates a signature of the input image.

[0261] A step 8214 of comparing the signature of the input image with the signature of the conceptual structure.

[0262] Step 8216 determines whether the signature of the input image matches any of the signatures of the conceptual structure based on signature matching criteria, where each signature of the conceptual structure is associated within a signature matching criterion determined based on the object detection parameters of the signature.

[0263] Step 8218 concludes that the input image contains an object related to the conceptual structure based on the determination result.

[0264] The matching criterion for a signature can be the minimum number of identifiers that show a match. For example, given a signature that contains dozens of identifiers, the minimum number can vary between a single identifier to all identifiers in the signature.

[0265] It should be noted that an input image may contain multiple objects, and the signature of the input image may match multiple cluster structures. The method 8200 is applicable to all matching processes, and the signature matching criteria can be set for each signature of each cluster structure.

[0266] Step 8210 may precede step 8202 of determining each signature matching criterion by evaluating the object detection ability of the signatures under different signature matching criteria.

[0267] Step 8202 may include:

[0268] A step 8203 of receiving or generating a signature for a set of test images.

[0269] For each of the different signature matching criteria, calculate 8204 the object detection capability of the signature.

[0270] Step 8206 of selecting a signature matching criterion based on the object detection ability of the signature under different signature matching criteria.

[0271] Object detection performance can reflect the percentage of signatures in a set of test images that match the signature.

[0272] Selecting the signature matching criterion includes selecting a signature matching criterion that, once applied, causes the percentage of signatures in the test image set that match the signature to close to a predefined desired percentage of signatures in the test image set that match the signature.

[0273] Object detection performance may reflect significant changes in the percentage of a signature in a set of test images that match the signature. For example, assume that the signature matching criterion is a minimum number of identifier matches, and changing the value of the minimum number of matches changes the match rate of test images. A significant change in percentage (e.g., a change of 10%, 20%, 30%, 40% or more) may indicate a desired value. The desired value may be set before a substantial change, near a substantial change, etc.

[0274] References in this specification to a method should be applied mutatis mutandis to a system capable of performing the method, and also to a non-transitory computer-readable medium storing instructions that, once executed by a computer, result in the performance of the method.

[0275] References in this specification to systems and other components should be applied mutatis mutandis to methods that may be performed by the systems, and to non-transitory computer-readable media storing instructions that may be executed by the systems.

[0276] References in this specification to a non-transitory computer readable medium should be applied mutatis mutandis to a system capable of executing instructions stored on the non-transitory computer readable medium, and should be applied mutatis mutandis to a method that can be executed by a computer reading instructions stored on the non-transitory computer readable medium.

[0277] Any combination of modules or units described in any of the figures, any part of this specification and / or in the claims may be provided. In particular, any combination of claimed features may be provided.

[0278] References to the terms "comprising" or "having" should also be construed as a reference to "consisting of" or "essentially consisting of." For example, a method comprising certain steps may include additional steps, may be limited to those particular steps, or may include additional steps that do not materially affect the basic and novel characteristics of the method.

[0279] The invention may also be embodied in a computer program for running on a computer system, comprising at least code portions for performing the steps of the method according to the invention when the computer program is run on a programmable device such as a computer system, or for enabling a programmable device to perform the functions of the device or system according to the invention. The computer program may be adapted to assign disk drives in the storage system to disk drive groups.

[0280] A computer program is a listing of instructions for a particular application program or operating system, etc. A computer program may include, for example, one or more of the following: subroutines, functions, procedures, object methods, object implementations, executable applications, applets, servlets, source code, object code, shared libraries / dynamic load libraries, and / or other sequences of instructions designed for execution on a computer system.

[0281] The computer program may be stored internally in a computer program product, such as a non-transitory computer readable medium. All or part of the computer program may be provided on a non-transitory computer readable medium that is permanently, removably, or remotely connected to an information processing system. The non-transitory computer readable medium may be any of the following, by way of non-limiting example: magnetic storage media, including disk and tape recording media; optical storage media, such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; non-volatile memory storage media, including semiconductor-based memory units, such as FLASH memory, EEPROM, EPROM, ROM, etc.; ferromagnetic digital memory; MRAM; volatile storage media, including registers, buffers or caches, main memory, RAM, etc. A computer process typically includes an executing (running) program or part of a program, current program values ​​and state information, and resources used by the operating system to manage the execution of the process. An operating system (OS) is software that manages the sharing of a computer's resources and provides a programmer with an interface to access those resources. The operating system processes system data and user input, and responds by allocating and managing tasks and internal system resources as a service to the system's users and programs. A computer system may include, for example, at least one processing unit, associated memory, and a number of input / output (I / O) devices. When executing a computer program, the computer system processes information according to the computer program and, as a result, generates information that is output via the I / O devices.

[0282] Although the invention has been described hereinabove with reference to specific embodiments thereof, it will be apparent, however, that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0283] Additionally, terms such as "front," "back," "top," "bottom," "upper," "lower," and the like in this specification and claims are used for descriptive purposes and not necessarily to describe permanent relative positions, with it being understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are operable, for example, in orientations other than those illustrated or otherwise described herein.

[0284] Those skilled in the art will recognize that the boundaries between logical blocks are merely exemplary, and that alternative embodiments may combine logical blocks or circuit elements or impose alternative decompositions of functionality on the various logical blocks or circuit elements. It should therefore be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures may be implemented which achieve the same functionality.

[0285] An arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, for purposes of this specification, two components that combine to achieve a particular functionality may be considered to be "associated" with one another such that the desired functionality is achieved, regardless of architecture or intervening components. Similarly, two components that are so associated may also be considered to be "operably connected" or "operably coupled" with one another to achieve the desired functionality.

[0286] Moreover, those skilled in the art will recognize that the above-described boundaries of operations are merely illustrative. Operations may be combined into a single operation, a single operation may be distributed among additional operations, or operations may be performed with at least partial overlap in time. Additionally, alternative embodiments may include multiple instances of a particular operation, and in various other embodiments, the order of operations may be changed. Also, for example, in one embodiment, the illustrated examples may be implemented as circuits located on a single integrated circuit or within the same device. Alternatively, the examples may be implemented as any number of separate integrated circuits or separate devices interconnected with each other in any suitable manner.

[0287] Also, for example, embodiments or portions thereof may be implemented as a soft or code representation of a physical circuit, or a logical representation that can be translated into a physical circuit, such as any suitable type of hardware description language.

[0288] Furthermore, the present invention is not limited to physical devices or units implemented in non-programmable hardware, but may also be applied to programmable devices or units that can perform desired device functions by operating in accordance with appropriate program code, such as mainframes, minicomputers, servers, workstations, personal computers, notepads, personal digital assistants, electronic games, automobiles and other embedded systems, mobile phones, and various other wireless devices (collectively referred to herein as "computer systems").

[0289] However, other modifications, variations, and alternatives are possible, and the specification and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

[0290] In the claims, any reference signs enclosed in parentheses shall not be construed as limiting the scope of the claims. The word "comprising" does not exclude the presence of elements or steps other than those recited in the claims. Furthermore, as used herein, the terms "a" or "an" are defined as one or more. Furthermore, the use of introductory phrases such as "at least one" and "one or more" in the claims shall not be construed as implying that the introduction of another claim element by the indefinite article "a" or "an" limits a particular claim containing such introduced claim element to an invention containing only one such element, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and an indefinite article such as "a" or "an". The same applies to the use of definite articles. Unless otherwise noted, terms such as "first" and "second" are used to arbitrarily distinguish between elements that such terms describe. Thus, these terms are not necessarily intended to indicate a chronological or other priority of such elements.The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

[0291] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art, and it is therefore to be understood that the appended claims are intended to cover all such modifications and changes which fall within the true spirit of the invention.

Claims

1. A method for providing a pedestrian step-down warning without restrictions, The vehicle's computerized system receives an unregulated pedestrian step-down (PSD) indicator, which indicates an unregulated PSD situation. To acquire information perceived regarding the environment of the aforementioned vehicle, Processing the perceived information, the processing includes searching for at least one unregulated PSD index, The vehicle autonomously determines, upon discovering at least one unregulated PSD identifier, that it is approaching a situation in which a pedestrian is expected to step down into the vehicle's drivable space within the vehicle's environment, and that this autonomous determination triggers a response to the discovery, the response being an immediate response, and performed by at least the vehicle's computerized system. A method of including.

2. The aforementioned unregulated PSD index is To get the first group of untagged videos, Finding unrestricted PSD events within the aforementioned first group of untagged videos, The generation process includes processing pre-PSD event video segments to generate the unrestricted PSD index. Generated by, The method according to claim 1, wherein each unrestricted PSD event video segment begins at a predetermined time interval prior to the corresponding unrestricted PSD event among the unrestricted PSD events.

3. The method according to claim 2, wherein the generation of the unrestricted PSD index comprises removing identifiers of objects that appear in the prior unrestricted PSD event video segment but are not features of the unrestricted PSD event.

4. The method according to claim 3, wherein the removal includes searching for objects that appear in a video segment different from the PSD event video segment that is not subject to prior restrictions, in a manner exceeding a predefined probability.

5. The generation of the PSD index without the aforementioned restrictions is, To generate signatures for PSD event video segments without prior restrictions, To cluster the signatures of the aforementioned PSD event video segments without prior restrictions and provide multiple clusters, To generate cluster signatures for the aforementioned multiple clusters and The method according to claim 2, which includes the method described in claim 2.

6. A non-temporary computer-readable medium for an uncontrolled pedestrian step-down warning, wherein the non-temporary computer-readable medium is The vehicle's computerized system receives an unregulated pedestrian step-down (PSD) indicator, which shows unregulated PSD situations. Information is obtained regarding the environment of the aforementioned vehicle. Processing the perceived information, the processing includes searching for at least one unregulated PSD index. The vehicle autonomously determines, upon discovering at least one unregulated PSD index, that it is approaching a situation in which a pedestrian is expected to step down into the vehicle's drivable space within the vehicle's environment, wherein the autonomous determination triggers a response to the discovery, the response being an immediate response, and a non-temporary computer-readable medium storing instructions to be executed by at least the vehicle's computerized system.

7. The aforementioned unregulated PSD index is To get the first group of untagged videos, Finding unrestricted PSD events within the aforementioned first group of untagged videos, To generate the unrestricted PSD index by a generation process that includes processing the unrestricted PSD event video segment. Generated by, The non-temporary computer-readable medium according to claim 6, wherein each unrestricted PSD event video segment begins at a predetermined time interval prior to the corresponding unrestricted PSD event among the unrestricted PSD events.

8. The generation of the unregulated PSD index comprises removing identifiers of objects that appear in the video segment of the prior unregulated PSD event but are not features of the unregulated PSD event, according to claim 7, for a non-temporary computer-readable medium.

9. The non-temporary computer-readable medium according to claim 8, wherein the removal includes searching for objects that appear in a video segment different from the PSD event video segment that is not subject to prior restrictions, in a manner exceeding a predefined probability.

10. The generation of the PSD index without the aforementioned restrictions is, To generate PSD event video segment signatures without prior restrictions, To cluster the aforementioned PSD event video segment signatures, which are not subject to prior restrictions, and to provide multiple clusters, To generate cluster signatures for the aforementioned multiple clusters and A non-temporary computer-readable medium according to claim 7, comprising the above.

Images