Object position coordinate determination

Abstract

A system includes a processor and a memory. The memory stores instructions executable by the processor to receive an image from a fixed camera. The memory stores instructions for determining position coordinates of an object identified in the image based on position coordinates specified for the image. The memory stores instructions for operating a vehicle based on the object position coordinates.

Description

Technical Field

[0001] The present invention relates to a system including a processor and a memory. The memory stores instructions executable by the processor to receive an image from a fixed camera. The memory stores instructions for determining position coordinates of an object identified in the image based on position coordinates specified for the image. The memory stores instructions for operating a vehicle based on the object position coordinates. Background Art

[0002] Vehicle collision avoidance systems may use vehicle sensor data and / or broadcast messages from other vehicles to identify objects that present a collision risk and determine the location of the identified objects. However, there is the difficulty that certain objects may be difficult to identify and / or do not broadcast messages from which a collision risk may be identified. Additionally, it is a difficulty that current collision avoidance techniques rely on techniques that are often expensive and / or difficult to implement. Therefore, it would be beneficial to propose new technical solutions for identifying collision risk objects for vehicles that effectively and efficiently provide collision avoidance for objects that are not adequately addressed by current techniques. Summary of the invention

[0003] A system includes a processor and a memory storing instructions executable by the processor to receive an image from a fixed camera, determine position coordinates of an object identified in the image based on position coordinates specified for the image, and operate a vehicle based on the object position coordinates.

[0004] The system may include instructions for determining the object position coordinates based on the stored position coordinates of each of a plurality of reference points in the image.

[0005] The system may also include instructions for determining the object position coordinates by identifying one or more projection points associated with the object in the image, and determining the object position coordinates based on the stored position coordinates.

[0006] The system, wherein the location coordinates specified for the image can be associated with a visual feature in the received image, the visual feature being one of a road intersection, a building, a traffic sign, and a calibration sign.

[0007] The system may also include instructions for receiving geometric properties of the visual feature and position coordinates of the visual feature, detecting the visual feature in a received camera image based on the received geometric properties, and specifying the position coordinates for the image by associating the received position coordinates with a projection point of the visual feature in the received image.

[0008] The system, wherein the calibration marker is a portable marker, and the calibration marker further comprises a GPS sensor and a second processor programmed to transmit location coordinates of the calibration marker based on data received from the GPS sensor of the calibration marker.

[0009] The system may also include instructions for adjusting predetermined position coordinates of a reference position upon determining that a projection point associated with the reference position in the image has moved, and determining the object position coordinates based on the adjusted position coordinates of the reference position.

[0010] The system may also include instructions for determining location coordinates specified for the image based on the camera position and the camera orientation.

[0011] The system may also include instructions for determining at least one of a trajectory and a velocity of the object based on the received camera image.

[0012] The system may also include instructions for operating the vehicle by actuating a vehicle brake actuator based on the detected position coordinates of the object and the trajectory of the vehicle.

[0013] A method comprising: receiving an image from a fixed camera; determining position coordinates of an object identified in the image based on position coordinates specified for the image; and

[0014] A vehicle is operated based on the object position coordinates.

[0015] The method may also include determining the object position coordinates based on the stored position coordinates of each of a plurality of reference points in the image.

[0016] The method may also include determining the object position coordinates by identifying one or more projection points associated with the object in the image, and determining the object position coordinates based on the stored position coordinates.

[0017] The method, wherein the location coordinates specified for the image may be associated with a visual feature in the received image, the visual feature being one of a road intersection, a building, a traffic sign, and a calibration sign.

[0018] The method may also include: receiving geometric properties of the visual feature and position coordinates of the visual feature; detecting the visual feature in the received camera image based on the received geometric properties; and specifying the position coordinates for the image by associating the received position coordinates with a projection point of the visual feature in the received image.

[0019] The method may further include transmitting the location coordinates of the calibration marker based on data received from a GPS sensor included in the calibration marker, wherein the calibration marker is a portable marker.

[0020] The method may further include adjusting predetermined position coordinates of the reference position upon determining that a projection point associated with the reference position in the image has moved, and determining the object position coordinates based on the adjusted position coordinates of the reference position.

[0021] The method may also include determining location coordinates specified for the image based on the camera position and the camera orientation.

[0022] The method may also include determining at least one of a trajectory and a velocity of the object based on the received camera image.

[0023] The method may further include performing a vehicle operation by actuating a vehicle brake actuator based on the detected position coordinates of the object and the trajectory of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an illustration of an example camera directed toward an example intersection.

[0025] Figure 2 Example stored position coordinates are shown superimposed on an example camera image.

[0026] Figure 3 is Figure 1 Example image received at the camera.

[0027] Figure 4 is a flow chart for determining the position coordinates of an object in the field of view of a camera.

[0028] Figure 5 is a flow chart of an exemplary process for calibrating a camera. DETAILED DESCRIPTION

[0029] introduction

[0030] A system including a processor and a memory is disclosed herein. The memory stores instructions executable by the processor to receive an image from a fixed camera. The memory stores instructions executable by the processor to perform the following: determine the position coordinates of an object identified in the image based on the position coordinates specified for the image, and operate a vehicle based on the object position coordinates.

[0031] The instructions may also include instructions for determining the object position coordinates based on the stored position coordinates of each of the plurality of reference points in the image.

[0032] The instructions may also include instructions for determining object position coordinates by identifying one or more projection points associated with the object in the image, and determining the object position coordinates based on the stored position coordinates.

[0033] The location coordinates specified for the image may be associated with a visual feature in the received image, the visual feature being one of a road intersection, a building, a traffic sign, and a calibration sign.

[0034] The instructions may also include instructions for receiving geometric properties of a visual feature and position coordinates of the visual feature, detecting the visual feature in a received camera image based on the received geometric properties, and specifying the position coordinates for the image by associating the received position coordinates with a projection point of the visual feature in the received image.

[0035] The calibration marker may be a portable marker, and the calibration marker may further include a GPS sensor and a processor programmed to transmit location coordinates of the calibration marker based on data received from the GPS sensor of the calibration marker.

[0036] The instructions may also include instructions for adjusting predetermined position coordinates of the reference position upon determining that a projection point associated with the reference position in the image has moved, and determining the object position coordinates based on the adjusted position coordinates of the reference position.

[0037] The instructions may also include instructions for determining location coordinates specified for the image based on the camera position and the camera orientation.

[0038] The instructions may also include instructions for determining at least one of a trajectory and a velocity of the object based on the received camera image.

[0039] The instructions may also include instructions for performing a vehicle operation by actuating a vehicle brake actuator based on the detected position coordinates of the object and the trajectory of the vehicle.

[0040] Also disclosed herein is a method that includes: receiving an image from a fixed camera; determining position coordinates of an object identified in the image based on position coordinates specified for the image; and operating a vehicle based on the object position coordinates.

[0041] The method may also include determining the object position coordinates based on the stored position coordinates of each of the plurality of reference points in the image.

[0042] The method may also include determining the object position coordinates by identifying one or more projection points associated with the object in the image, and determining the object position coordinates based on the stored position coordinates.

[0043] The location coordinates specified for the image may be associated with a visual feature in the received image, the visual feature being one of a road intersection, a building, a traffic sign, and a calibration sign.

[0044] The method may also include receiving geometric properties of the visual feature and position coordinates of the visual feature, detecting the visual feature in the received camera image based on the received geometric properties, and specifying the position coordinates for the image by associating the received position coordinates with a projection point of the visual feature in the received image.

[0045] The method may further include transmitting location coordinates of the calibration marker based on data received from a GPS sensor included in the calibration marker, wherein the calibration marker is a portable marker.

[0046] The method may further include adjusting predetermined position coordinates of the reference position upon determining that a projection point associated with the reference position in the image has moved, and determining the object position coordinates based on the adjusted position coordinates of the reference position.

[0047] The method may also include determining location coordinates specified for the image based on the camera position and the camera orientation.

[0048] The method may also include determining at least one of a trajectory and a velocity of the object based on the received camera image.

[0049] The method may also include performing a vehicle operation by actuating a vehicle brake actuator based on the detected position coordinates of the object and the trajectory of the vehicle.

[0050] Also disclosed is a computing device programmed to perform any of the above method steps.

[0051] A computer program product is also disclosed, the computer program product comprising a computer readable medium storing instructions executable by a computer processor to perform any of the above method steps.

[0052] System components

[0053] Figure 1 Camera 100 is shown with field of view 130, which is oriented toward an example area including intersection 120. Camera 100 is an electronic visual sensor that provides image data of field of view 130 of camera 100. Camera 100 may include a visual image sensor or a combination of visual and infrared image sensors. Image data, i.e., digital data storable in a memory in a predetermined format, may include, for example, pixels and a specified value associated with an image attribute of each pixel (such as color, light intensity, etc.). In one example, processor 110 may generate image data, such as image 200 (see Figure 2 ), which includes a visual representation of a field of view 130, such as an intersection 120 including roads 180a, 180b, a host vehicle 160, target objects such as buildings, pedestrians, traffic light signs, etc. (collectively referred to herein as objects 140), and / or a subject vehicle 161. Figure 1 In the example shown, the camera 100 is oriented toward the center 185 of the intersection 120. The camera 100 may be mounted in various locations such as parking lots, crosswalks, airports, etc., and is typically oriented in a predetermined direction, i.e., parallel to and through the lens 230 of the camera 100 (see FIG. Figure 2 ) has an axis located at a specified height from the ground and is oriented relative to a vertical axis and each of two horizontal axes (e.g., determined relative to a map direction such as north, south, west, and east) to collect images 200 in a specified field of view 130.

[0054] The camera 100 is typically a conventional digital camera and may include, for example, a processor 110, an image sensor, and optical components. The camera 100 may output image data based on an optical signal (i.e., light) detected by the image sensor. The processor 110 may be programmed to detect an object in the received image 200 and determine the position coordinates, speed, etc. of the object (e.g., vehicle 160) in the received image 200, as described below with reference to Figure 2-Figure 3 discussed.

[0055] The vehicles 160, 161 may be powered in a variety of known ways (e.g., with electric motors and / or internal combustion engines). The vehicles 160, 161 may have common elements including computers 171, actuators 172, sensors 173, etc., as described below. The subject vehicle 160 and subject vehicles 161 are explicitly referred to herein to capture the distinction between the subject vehicle 160 with respect to which collision risk is assessed and avoided as disclosed herein and one or more subject vehicles 161, with the subject vehicles 161 and other target objects 140 being discussed as target objects with respect to which collision risk may be assessed and avoided for the subject vehicle 160.

[0056] The computer 171 includes a processor and a memory such as are known. The memory includes one or more forms of computer-readable media and stores instructions that can be executed by the computer for performing various operations (including operations as disclosed herein). The computer 171 can operate the vehicles 160, 161 in autonomous, semi-autonomous, or non-autonomous modes. For the purposes of this disclosure, the autonomous mode is defined as a mode in which the computer controls each of the propulsion, braking, and steering of the vehicles 160, 161; in the semi-autonomous mode, the computer controls one or both of the propulsion, braking, and steering of the vehicles 160, 161; in the non-autonomous mode, the human operator controls the propulsion, braking, and steering of the vehicles 160, 161.

[0057] The computer 171 may include programming to operate one or more of braking, propulsion (e.g., controlling acceleration of the vehicles 160, 161 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, etc.

[0058] The vehicles 160, 161 may include one or more sensors, such as cameras, radars, global positioning system (GPS) devices, etc. For example, the computer 171 may be programmed to broadcast the location coordinates of the vehicles 160, 161 based on data received from the GPS sensors included in the vehicles 160, 161. For example, the computer 171 may be programmed to actuate the brake actuator 172 of the vehicle 161 when it is determined that the distance between the vehicles 160, 161 is less than a predetermined threshold (e.g., 10 meters) based on the received location coordinates of the vehicle 160.

[0059] Vehicles 160, 161 may include actuators 172, which are implemented via circuits, chips or other electronic components that can actuate various vehicle subsystems according to known appropriate control signals. Actuators 172 can be used to control braking, acceleration and steering of vehicles 160, 161.

[0060] The computer of the vehicle 160, 161 may include or be communicatively coupled (e.g., via a vehicle network such as a communication bus described further below) to one or more processors, such as controllers included in the vehicle for monitoring and / or controlling various vehicle controllers, such as a powertrain controller, a brake controller, a steering controller, etc. The computer 171 is generally arranged to communicate over a vehicle communication network, such as a bus in the vehicle, such as a controller area network (CAN) or the like.

[0061] In addition, the computer 171 can be programmed to communicate with the vehicle 161, the mobile device, the camera 100, and / or with devices such as traffic lights via a wireless communication network. The wireless communication network (which may include a vehicle-to-vehicle (V-to-V) and / or vehicle-to-infrastructure (V-to-I) communication network) includes one or more structures by which the vehicles 160, 161, the camera 100, the mobile device, etc. can communicate with each other, the one or more structures including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or multiple topologies when multiple communication mechanisms are utilized). Exemplary V-to-V or V-to-I communication networks include cellular, Bluetooth, IEEE 802.11, dedicated short range communications (DSRC), cellular V2X, and / or wide area networks (WANs) that provide data communication services, the wide area network including the Internet.

[0062] The computer 171 can actuate the vehicles 160, 161 based on the location and / or direction of movement of other objects, such as pedestrian objects 140, other vehicles 160, 161, bicycles, etc. For example, by actuating the brake actuators 172 of the vehicles 160, 161 to brake. The computer 171 can be programmed to receive location coordinates, such as global positioning system (GPS) coordinates, from another vehicle 161, etc. However, some objects 140, such as pedestrian objects 140, bicycles, etc., may lack a device with a GPS sensor to transmit their location. Reference Figure 1-Figure 2 , the processor 110 and / or the host vehicle 160 computer 171 may be programmed to receive the image 200 from the fixed camera 100. The processor 110 may determine the position coordinates of the object 140 (e.g., pedestrian object 140, object vehicle 161, etc.) identified in the image 200 based on the position coordinates specified for the image 200. In one example, the computer 171 of the vehicle 160 may also be programmed to operate the vehicle 160 based on the determined position coordinates of the object 140.

[0063] "Fixed" is intended herein to have its simple and ordinary meaning, i.e., not moving or fixed in a certain location, such as on a pole. Thus, the field of view 130 of a fixed camera 100 generally does not change after the camera 100 has been installed (e.g., at an intersection 120), unless the position and / or orientation of the fixed camera 100 is changed relative to the surrounding environment, such as by a technician.

[0064] In the present disclosure, "position coordinates" means coordinates that specify a position on the ground, for example, GPS position coordinates. For example, the position coordinates may be determined in a reference multidimensional Cartesian coordinate system having a predetermined origin (for example, on the ground). For example, the position coordinates may include X, Y, Z coordinates. X, Y, and Z may represent the longitudinal, lateral, and height coordinates of the position, respectively. Specifying position coordinates for an image means specifying position coordinates for a position included in the image 200 received at the fixed camera 100. Reference Figure 2 Specifying location coordinates for an image is discussed in more detail.

[0065] Processor 110 may be programmed to perform operations of vehicle 160 , such as steering, braking, and / or propulsion, by actuating brake actuator 172 of vehicle 160 based on the detected position coordinates of object 140 (eg, object vehicle 161 ) and trajectory t1 of object vehicle 161 .

[0066] Trajectory, such as trajectory t1, t2, t3, in the context of the present disclosure refers to the actual, expected and / or projected movement path of an object, such as object vehicle 161, pedestrian object 140, bicycle object 140, etc., starting from the current position of object 140 and extending at least a predetermined distance, such as 5 meters, in front of object 140. Computer 171 of host vehicle 160 can be programmed to actuate actuators 172 of host vehicle 160 based on the determined trajectory, such as by accelerating, braking and / or steering to navigate host vehicle 160. The corresponding trajectory of detected object 140 can include curves and / or straight lines on the ground, such as roads 180a, 180b, sidewalks, etc.

[0067] The processor 110 may be programmed to determine the trajectory and / or velocity of the object 140 based on the received camera 100 images 200. For example, the processor 110 may be programmed to determine the trajectory of the object 140 by periodically determining the position of the object 140 to plot the actual trajectory of the object 140. An extrapolation technique may then be used to determine an expected or projected trajectory based on the plotted actual (e.g., historical to current moment) trajectory.

[0068] The processor 110 may be programmed to classify the objects 140 based on a risk level and to perform operations of the vehicle 160 based on the risk level associated with the identified object category. The risk level is an assigned value within a predetermined range (e.g., from 1 to 10, or "high", "medium", and "low", etc.). The risk level indicates the risk of causing damage to humans and / or the vehicle 160. As shown in the example Table 1, humans and / or large objects (e.g., buildings) may be assigned a higher risk level because a collision with such objects 140 may cause greater damage to the vehicle 160 and / or humans than an object with a low risk level (such as a pothole or bump in the road). The computer 171 of the vehicle 160 may be programmed to cause actions, such as steering, accelerating, and / or braking, to prevent a collision with the object 140 based on the actions associated with the determined risk level, for example, as shown in Table 1.

[0069]

[0070] Table 1

[0071] exist Figure 2 In one example shown, the position coordinates of various points (referred to herein as reference points 210) within the field of view 130 of the fixed camera 100 may be specified. Although for ease of description, Figure 2The reference points 210 are shown as being uniformly distributed, but the reference points 210 may be distributed in any other form, for example, some or all may be randomly or randomly distributed. The processor 110 may be programmed to determine the position coordinates of, for example, the object vehicle 161, the pedestrian object 140, etc. based on the stored position coordinates of each of the reference points 210 drawn in the image 200. The position coordinates of the reference points 210 may be stored in a memory of the processor 110, for example, as represented in Table 2. Here, the term "reference point" refers to a point (e.g., a pixel) in the received image 200 that can be specified by a height and width coordinate in the image 200 (e.g., in a Cartesian two-dimensional coordinate system with a predetermined origin (such as the corner point 240)). The position coordinates of a point (e.g., reference point 210d) include the X', Y' coordinates of the point relative to the origin (such as the corner point 240). X' and Y' may represent the longitudinal and lateral coordinates of a point in the image 200 position, respectively. In this context, a "reference position" refers to a point on the ground, such as a road, a sidewalk, a building, an object, etc. The reference position 220 may be projected on the reference point 210d (or area) in the received image 200. For the reference position 220 to correspond to or be projected to the point 210d, it means that the line 250 passing through the reference point 210d and the lens 230 of the camera 100 also includes the reference position 220 (and vice versa). For example, the projection of the object vehicle 161 is the projection area 260 in the image 200. As another example, the projection of the pedestrian object 140 is the object projection area 270.

[0072]

[0073] Table 2

[0074] Processor 110 may be programmed to detect an object 140 (e.g., a pedestrian object 140, an object vehicle 161, etc.) in image 200 and determine the position coordinates of object 140 by identifying one or more projection points (e.g., projection areas 260, 270) associated with the detected object 140 in image 200. Processor 110 may also be programmed to determine the position coordinates of object 140 by referring to a plurality of projection points such as projection areas 260, 270. Figure 1The stored data shown in the figure, based on the stored position coordinates of the plurality of reference points 210, for example, determines the position coordinates of the object vehicle 161. The processor 110 may be programmed to identify three or more reference points 210a, 210b, 210c, 210d that are adjacent to or overlap the projection area 260 of the vehicle 160 in the image, and determine the position coordinates of the vehicle 160 using triangulation, trilateration and / or interpolation techniques based on the identified reference points 210 adjacent to the projection area 260. In this context, overlapping means that the reference point 210 is on the projection area 260 of the object vehicle 161; adjacent means that the reference point 210 is closer to the projection area 260 than any other reference point 210. In one example, the processor 110 may be programmed to determine the position coordinates of each or a selected set of the image 200 data points, for example, the position coordinates of each pixel, based on the stored position coordinates (e.g., Table 2), and store the position coordinates associated with each of the pixels of the image 200 in the memory of the processor 110. For example, 5 (five) reference points 210 are stored, and processor 110 determines a corresponding reference position associated with each pixel of image 200 based on the five stored position coordinates. In this example, because the position coordinates of each pixel or a selected set of pixels in image 200 are determined, processor 110 can be programmed to find the position coordinates of detected object 140.

[0075] As described above, the position coordinates of the plurality of reference points 210 may be stored in, for example, a memory of the camera 100. Figure 3 Various example techniques are described for determining the position coordinates of reference point 210. In the present context, determining the position coordinates of reference point 210 may be referred to as "calibration."

[0076] Figure 3 An example image 300 is shown received at a camera 100 installed at an intersection 120. In order to calibrate the camera 100, i.e., determine the position coordinates of the reference points 210 in the image 300, the processor 110 can be programmed to associate the reference points 210 in the image 300 with the position coordinates of the reference positions projected on the respective reference points 210. In one example, the reference positions can be associated with visual features (e.g., objects 140, calibration signs 190, etc.) in the received image 300. In other words, by detecting visual features (such as traffic sign objects 140) in the received image 300, the reference points 210 in the image 300 (i.e., points within the projected areas 260, 270 of the visual features) can be determined and can be associated with, for example, GPS position coordinates of, for example, the object 140, the object vehicle 161, etc.

[0077] The processor 110 may be programmed to receive, for example, from a remote computer, a memory of the processor 110, etc., geometric properties of a visual feature (e.g., a traffic sign object 140) and position coordinates of the visual feature to detect the visual feature in the received image 300 based on the received geometric properties. The geometric properties may include information such as size, shape, pattern, etc. of the object 140, the road 180a, 180b, etc. The processor 110 may be programmed to store the received position coordinates as designated position coordinates for the image 300 by associating the received position coordinates with, for example, the projection area 260, 270 position of the visual feature in the received image 300. The processor 110 may be programmed to receive, for example, the geometric properties of the visual feature from a remote computer and detect the visual feature, for example, the bicycle object 140, in the received image 300. Additionally or alternatively, the processor 110 may be programmed to store the geometric properties of the visual feature in the memory of the processor 110. The processor 110 may then determine a reference point of the projected areas 260, 270 in the image 300, such as a center, a corner, a peak, etc., such as the peak of the projected area 270. "Peak" refers to the highest point of an object above the ground. The processor 110 may then determine a reference point 210, such as the peak of the traffic sign object 140 in the image 300, having coordinates (X1', Y1'). The processor 110 may associate the determined reference point (X1', Y1') of the image 300 with the received GPS reference position (X1, Y1, Z1) of the traffic sign object 140. X1' and Y1' represent a position in the image 300, such as in pixels, while X1, Y1, and Z1 are position coordinates in a three-dimensional Cartesian coordinate system. Therefore, the processor 110 may store the position coordinates (X1, Y1, Z1) and the reference point (X1', Y1'), such as in the form of a table such as Table 2. Additionally or alternatively, a GPS position coordinate system may be used to determine a position on the ground, or the like, where a position may be determined by a pair of numbers specifying coordinates (eg, latitude and longitude).

[0078] Another example technique for calibrating the reference position of camera 100 may include using calibration markers 190 (see Figure 1). The calibration mark 190 may be a portable mark, and the calibration mark 190 may also include a GPS sensor and a computing device having a processor and a memory. The processor of the calibration mark 190 may be programmed to send the position coordinates of the calibration mark 190 via, for example, a wireless communication network based on data received from the GPS sensor of the calibration mark 190. The calibration mark 190 may have specific visual features, such as a printed pattern. For example, the processor 110 may be programmed to detect the bottom corner of the calibration mark projection area 270 in the image 300, for example, at a reference point (X2', Y2'), and receive the position coordinates (X2, Y2, Z2) of the calibration mark 190 via the wireless communication network. Therefore, advantageously, by moving the calibration mark 190 to multiple positions within the field of view 130 of the camera 100, three of the more reference points 210 may be determined.

[0079] As another example technique for calibrating the camera 100, the processor 110 may be programmed to determine position coordinates specified for the image 300 based on the position of the camera 100 and the orientation of the camera 100. For example, the processor 110 may be programmed to receive the position and orientation of the camera 100 from, for example, a remote computer and / or one or more internal sensors included in the housing of the camera 100. The processor 110 may be programmed to determine the position coordinates of one or more positions within the field of view 130 of the camera 100 based on the received position of the camera 100, the orientation of the camera 100, and / or the optical properties of the camera 100. The optical properties may include focus, field of view 130, distortion characteristics of the lens 230 of the camera 100, and the like.

[0080] As described above, the camera 100 is fixed, but the orientation and / or position of the camera 100 may change due to various reasons such as wear, wind, etc. The processor 110 may be programmed to adjust the predetermined position coordinates of the reference position 220 upon determining that the reference point 210 associated with the reference position in the image 200, 300 has moved. In this context, "movement" means a change in the position and / or orientation of the position 220 relative to the camera 100 at the time of calibration, i.e., determining the position coordinates of the reference point 210 in the image 200, 300 received by the camera 100. The processor 110 may also be programmed to determine the position coordinates of the object 140 based on the adjusted position coordinates of the reference position.

[0081] As an example, the processor 110 may be programmed to store information of a visual feature (e.g., a traffic sign object 140) as a reference location 220 and associate it with a projection area 270 of the traffic sign object 140 as the reference point 210. As long as the camera 100 is stationary and the traffic sign 140 is not unloaded and / or is not moving, the projection area 270 in the image 300 does not move. Therefore, the processor 110 may be programmed to determine whether the camera 100 has moved by comparing the current projection area 270 of the visual feature with the stored projection area 270. The processor 110 may store visual feature data, such as geometric property information of the traffic sign 140. Additionally, the processor 110 may be programmed to associate the stored visual feature geometric properties (e.g., shape, size, etc.) with the reference point 210. The processor 110 may be programmed to detect the traffic sign object 140 in the received image 300, for example, periodically, based on the stored visual feature data, and determine the location of the projection area 270 of the visual feature in the image 300. The processor 110 may be programmed to adjust the reference position upon determining that, for example, the peak of the traffic sign projection area 270 has moved more than a specified distance (e.g., one pixel) from the stored position. The processor 110 may be programmed to adjust the reference position 220 and / or the stored position coordinates of the reference point 210 (e.g., in a set of data such as shown in Table 2 above) based on the moved position of the reference point 210 in the image 300, for example, using triangulation techniques.

[0082] Additionally or alternatively, the processor 110 may be programmed to determine (and store) an adjusted reference position and / or reference point 210 based on the received position coordinates of the detected vehicle 160 and the detected vehicle 161. In other words, the computer 171 of the vehicle 160 may be programmed to determine the position of the vehicle 160 based on data received from the GPS sensor 173 of the vehicle 160 and broadcast the position coordinates of the vehicle 160. The processor 110 may be programmed to determine the position coordinates of the vehicle 160 in the received image 200 and store the position of the vehicle 160 in the image 200 and the associated GPS position coordinates received from the vehicle 160, for example, via a wireless communication network.

[0083] In another example, two or more cameras 100 may be installed in one area. As an example, the first camera 100 and the second camera 100 may be installed in Figure 1The first camera 100 may be configured to project a trajectory t1 of the object 140 based on a plurality of images including an image of the vehicle 160. In this example with multiple cameras 100, when the vehicle 160 leaves the field of view 130 of the first camera 100 and enters the field of view 130 of the second camera 100, for example, the processor 110 of the second camera 100 may be programmed to determine the trajectory t1 of the object vehicle 161 based on a combination of images from the first camera 100 and the second camera 100.

[0084] deal with

[0085] Figure 4 4 is a flow chart of an exemplary process 400 for determining position coordinates of an object 140 in the field of view 130 of the camera 100. For example, the processor 110 of the camera 100, a remote computer, the computer 171 of the vehicle 160, or a combination thereof may be programmed to perform the blocks of the process 400.

[0086] Process 400 begins at block 410 , where processor 110 receives image data from camera 100 .

[0087] In decision block 415, processor 110 determines whether object 140, vehicle object 161, etc. is detected. For example, processor 110 may be programmed to detect object 140 in the received image data using image processing techniques as described above. Processor 110 may be programmed to detect object 140 based at least in part on geometric properties of object 140 stored in a memory of processor 110 and / or received from a remote computer. If processor 110 determines that object 140 has been detected, process 400 proceeds to block 420; otherwise, process 400 returns to decision block 415.

[0088] In block 420, the processor 110 determines the position coordinates of the detected object 140 (e.g., the pedestrian object 140). The processor 110 may be programmed to determine the position coordinates of the object 140 by identifying the projection area 270 associated with the detected pedestrian object 140 in the image 300, and to determine the position coordinates of the pedestrian object 140 based on the stored reference position coordinates. The processor 110 may be programmed to determine the speed and / or trajectory t3 of the detected pedestrian object 140 based on the received image data and the stored position coordinates of the reference position, etc. Additionally, the processor 110 may be programmed to determine the risk category of the detected object based on a risk classification method, for example, Table 1.

[0089] Next, in box 430, the processor 110 operates the vehicle 160, for example, actuating the actuator 172 of the vehicle 160. In one example, the processor 110 can be programmed to perform a collision avoidance maneuver, including performing an operation of the vehicle 160 by sending instructions to the brake actuator 172 of the vehicles 160, 161 based on the position coordinates, speed and / or trajectory t3 of the detected object. In another example, the processor 110 sends the position coordinates, speed and / or trajectory t3 of, for example, a pedestrian object 140 to the vehicle 160. Thus, the computer 171 of the vehicle 160 can be programmed to perform an operation of the vehicle 160 based on the information received from the processor 110 of the camera 100. In yet another example, the processor 110 can be programmed to broadcast the position coordinates, speed and / or trajectory of the detected object 140 via a wireless communication network. Some objects 140 can broadcast their position coordinates, such as the object vehicle 161. In one example, the processor 110 may be programmed to operate the vehicle 160 only when it is determined based on information received via the wireless communication network that, for example, the detected bicycle object 140 is not broadcasting its location coordinates. Alternatively, the processor 110 and / or the vehicle 160 computer 171 may be programmed to operate the vehicle 160 based on the determined risk category of the object 140.

[0090] Next, in decision block 440, processor 110 determines whether camera 100 has moved since the position coordinates of reference point 210 were stored. Processor 110 may be programmed to determine that camera 100 has moved when it determines that at least the projection of a visual feature associated with reference point 210 in the received image has moved compared to the stored position of the projection. If processor 110 determines that camera 100 has moved, process 400 proceeds to block 445; otherwise, process 400 ends, or alternatively, returns to decision block 405, but in the event that ... Figure 4 Not shown.

[0091] In block 445, processor 110 adjusts the position coordinates of reference point 210. Processor 110 may be programmed to determine the movement of camera 100 and adjust the reference position associated with reference point 210 (e.g., the position coordinates of reference position 220) based on the determined movement of camera 100. Processor 110 may be programmed to store the adjusted reference position and / or reference point 210 in, for example, a memory of camera 100. Additionally or alternatively, processor 110 may be programmed to store the adjusted reference position and / or reference point 210 based on the received position coordinates of detected vehicle 160 and detected vehicle 161.

[0092] After block 445, process 400 ends, or optionally returns to decision block 405, but is not described here for ease of illustration. Figure 4 Shown in.

[0093] Figure 5 is a flow chart of an exemplary process 500 for calibrating camera 100. For example, processor 110 of camera 100 may be programmed to perform the blocks of process 500.

[0094] Process 500 begins at decision block 510, where processor 110 determines whether the geometric properties of the visual feature have been received and / or stored. Processor 110 may be programmed to receive the visual feature geometric properties (which include size, shape, etc.) and position coordinates from a remote computer, for example, via a wireless communication network. The visual feature may be a calibration mark 190, a traffic sign object 140, etc. If processor 110 determines that the visual feature data has been received and / or stored, process 500 proceeds to decision block 520; otherwise, process 500 returns to decision block 510.

[0095] In decision block 520, processor 110 determines whether a visual feature is detected. Processor 110 may be programmed to detect a visual feature, such as calibration mark 190, based on received visual feature geometry. For example, processor 110 may be programmed to detect calibration mark 190 based on received size, shape, etc. of calibration mark 190. If processor 110 determines that a visual feature has been detected, process 500 proceeds to block 530; otherwise, process 500 returns to decision block 520.

[0096] In block 530, processor 110 stores the received position coordinates of reference point 210. For example, processor 110 may be programmed to store the position coordinates of projection area 270 and visual features in a table such as presented in Table 2 in a memory of processor 110. Additionally, the processor may be programmed to store geometric properties of visual features. Thus, advantageously, if camera 100 moves, processor 110 may be programmed to detect visual features and adjust the stored table based on the stored geometric properties of the visual features, as described above.

[0097] After block 530, process 500 ends, or optionally returns to decision block 510, but Figure 5 Not shown.

[0098] The computing devices discussed herein typically each include instructions that can be executed by one or more computing devices (such as those identified above) and used to perform the blocks or steps of the above-described processes. The computer-executable instructions can be compiled or interpreted by a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, Java, PHP, and / or other programming languages. TM , C, C++, Visual Basic, Java Script, Perl, HTML, etc. In general, a processor (e.g., a microprocessor) receives instructions, such as from a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. A variety of computer-readable media can be used to store and transmit such instructions and other data. Files in a computing device are typically collections of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0099] Computer readable media include any media that participate in providing data (e.g., instructions) that can be read by a computer. Such media can take many forms, including but not limited to non-volatile media, volatile media, etc. Non-volatile media include, for example, optical disks or disks and other persistent memories. Volatile media include dynamic random access memory (DRAM), which typically constitutes a main memory. Common forms of computer readable media include, for example, floppy disks, floppy disks, hard disks, tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs, EPROMs, FLASHs, EEPROMs, any other memory chips or cassettes, or any other media that a computer can read.

[0100] With respect to the media, processes, systems, methods, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes can be practiced by performing the described steps in an order different from the order described herein. It should also be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the descriptions of systems and / or processes herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the disclosed subject matter.

[0101] Therefore, it should be understood that the present disclosure including the above description and the accompanying drawings and the following claims is intended to be illustrative and not restrictive. After reading the above description, many embodiments and applications other than the examples provided will be apparent to those skilled in the art. The scope of the present invention should not be determined with reference to the above description, but should be determined with reference to the claims attached and / or based thereon included in the non-provisional patent application together with the full scope of equivalents to which such claims are entitled. It is conceivable and expected that future developments will occur in the technology discussed herein, and the disclosed systems and methods will be incorporated into such future embodiments. In summary, it should be understood that the disclosed subject matter is capable of modification and variation.

Claims

1. A system for a vehicle, comprising a processor and a memory, the memory storing instructions executable by the processor to: receiving images from a fixed camera; determining the position coordinates of an object identified in the image based on position coordinates specified for the image, wherein the position coordinates specified for the image are associated with visual features in the received image, wherein the visual features include a calibration marker, wherein the calibration marker is a portable marker, and the calibration marker further includes a GPS sensor and a second processor programmed to transmit the position coordinates of the calibration marker based on data received from the GPS sensor of the calibration marker; Receiving the geometric properties of the visual feature and the position coordinates of the visual feature; detecting the visual feature in the received camera image based on the received geometric property; assigning position coordinates to the image by associating the received position coordinates with a projection point of the visual feature in the received image; and Based on the object position coordinates, a vehicle is operated.

2. The system of claim 1, wherein the instructions further comprise instructions for determining the object position coordinates based on the stored position coordinates of each of a plurality of reference points in the image.

3. The system of claim 2, wherein the instructions further comprise instructions for: determining the object location coordinates by identifying one or more projection points associated with the object in the image; and The object position coordinates are determined based on the stored position coordinates.

4. The system of claim 1, wherein the visual feature further comprises one of a road intersection, a building, and a traffic sign.

5. The system of claim 1, wherein the instructions further comprise instructions for: When it is determined that a projection point associated with a reference position in the image has moved, adjusting the predetermined position coordinates of the reference position; and Based on the adjusted position coordinates of the reference position, the object position coordinates are determined.

6. The system of claim 1, wherein the instructions further comprise instructions for determining the position coordinates specified for the image based on a camera position and an orientation of the camera.

7. The system of claim 1, wherein the instructions further comprise instructions for determining at least one of a trajectory and a velocity of the object based on the received camera image.

8. The system of claim 1, wherein the instructions further include instructions for operating the vehicle by actuating a vehicle brake actuator based on the position coordinates of the detected object and the trajectory of the vehicle.

9. A method for a vehicle, comprising: receiving images from a fixed camera; determining position coordinates of an object identified in the image based on position coordinates specified for the image, the position coordinates specified for the image being associated with visual features in the received image, wherein the visual features include calibration marks; transmitting the location coordinates of the calibration marker based on data received from a GPS sensor included in the calibration marker, wherein the calibration marker is a portable marker; Receiving the geometric properties of the visual feature and the position coordinates of the visual feature; detecting the visual feature in the received camera image based on the received geometric property; assigning position coordinates to the image by associating the received position coordinates with a projection point of the visual feature in the received image; as well as Based on the object position coordinates, a vehicle is operated.

10. The method of claim 9, further comprising determining the object position coordinates based on the stored position coordinates of each of a plurality of reference points in the image.

11. The method of claim 10, further comprising: determining the object position coordinates by identifying one or more projection points associated with the object in the image; as well as The object position coordinates are determined based on the stored position coordinates.

12. The method of claim 9, wherein the visual feature further comprises one of a road intersection, a building, and a traffic sign.

13. The method of claim 9, further comprising: When determining that a projection point associated with a reference position in the image has moved, adjusting a predetermined position coordinate of the reference position; as well as Based on the adjusted position coordinates of the reference position, the object position coordinates are determined.

14. The method of claim 9, further comprising determining the position coordinates specified for the image based on a camera position and an orientation of the camera.

15. The method of claim 9, further comprising determining at least one of a trajectory and a velocity of the object based on the received camera image.

16. The method of claim 9, further comprising performing the vehicle operation by actuating a vehicle brake actuator based on the position coordinates of the detected object and the trajectory of the vehicle.

Images